Max level shown:
1 [ Mar 7 22:33:43 Enabled. ]
2 [ Mar 7 22:33:43 Rereading configuration. ]
3 [ Mar 7 22:33:44 Rereading configuration. ]
4 [ Mar 7 22:33:50 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/omicron-nexus/bin/nexus /var/svc/manifest/site/nexus/config.toml &"). ]
5 [ Mar 7 22:33:50 Method "start" exited with status 0. ]
6 note: configured to log to "/dev/stdout"
72025-03-07T22:33:50.987ZDEBGnexus: registered DTrace probes
82025-03-07T22:33:50.989ZINFOnexus: setting up nexus server file = nexus/src/lib.rs:78
92025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = Action file = nexus/auth/src/authz/oso_generic.rs:68
102025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = AnyActor file = nexus/auth/src/authz/oso_generic.rs:68
112025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = AuthenticatedActor file = nexus/auth/src/authz/oso_generic.rs:68
122025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = BlueprintConfig file = nexus/auth/src/authz/oso_generic.rs:68
132025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = Database file = nexus/auth/src/authz/oso_generic.rs:68
142025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = DnsConfig file = nexus/auth/src/authz/oso_generic.rs:68
152025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = Fleet file = nexus/auth/src/authz/oso_generic.rs:68
162025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = Inventory file = nexus/auth/src/authz/oso_generic.rs:68
172025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = IpPoolList file = nexus/auth/src/authz/oso_generic.rs:68
182025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = ConsoleSessionList file = nexus/auth/src/authz/oso_generic.rs:68
192025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = DeviceAuthRequestList file = nexus/auth/src/authz/oso_generic.rs:68
202025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = SiloCertificateList file = nexus/auth/src/authz/oso_generic.rs:68
212025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = SiloIdentityProviderList file = nexus/auth/src/authz/oso_generic.rs:68
222025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = SiloUserList file = nexus/auth/src/authz/oso_generic.rs:68
232025-03-07T22:33:50.996ZINFOnexus (ServerContext): registering Oso class class = TargetReleaseConfig file = nexus/auth/src/authz/oso_generic.rs:68
242025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = Project file = nexus/auth/src/authz/oso_generic.rs:68
252025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = Disk file = nexus/auth/src/authz/oso_generic.rs:68
262025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = Snapshot file = nexus/auth/src/authz/oso_generic.rs:68
272025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = ProjectImage file = nexus/auth/src/authz/oso_generic.rs:68
282025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = AffinityGroup file = nexus/auth/src/authz/oso_generic.rs:68
292025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = AntiAffinityGroup file = nexus/auth/src/authz/oso_generic.rs:68
302025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = Instance file = nexus/auth/src/authz/oso_generic.rs:68
312025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = IpPool file = nexus/auth/src/authz/oso_generic.rs:68
322025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = InstanceNetworkInterface file = nexus/auth/src/authz/oso_generic.rs:68
332025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = Vpc file = nexus/auth/src/authz/oso_generic.rs:68
342025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = VpcRouter file = nexus/auth/src/authz/oso_generic.rs:68
352025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = InternetGateway file = nexus/auth/src/authz/oso_generic.rs:68
362025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = InternetGatewayIpPool file = nexus/auth/src/authz/oso_generic.rs:68
372025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = InternetGatewayIpAddress file = nexus/auth/src/authz/oso_generic.rs:68
382025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = RouterRoute file = nexus/auth/src/authz/oso_generic.rs:68
392025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = VpcSubnet file = nexus/auth/src/authz/oso_generic.rs:68
402025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = FloatingIp file = nexus/auth/src/authz/oso_generic.rs:68
412025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = Image file = nexus/auth/src/authz/oso_generic.rs:68
422025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = SiloImage file = nexus/auth/src/authz/oso_generic.rs:68
432025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = AddressLot file = nexus/auth/src/authz/oso_generic.rs:68
442025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = Blueprint file = nexus/auth/src/authz/oso_generic.rs:68
452025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = LoopbackAddress file = nexus/auth/src/authz/oso_generic.rs:68
462025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = Certificate file = nexus/auth/src/authz/oso_generic.rs:68
472025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = ConsoleSession file = nexus/auth/src/authz/oso_generic.rs:68
482025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = DeviceAuthRequest file = nexus/auth/src/authz/oso_generic.rs:68
492025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = DeviceAccessToken file = nexus/auth/src/authz/oso_generic.rs:68
502025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = PhysicalDisk file = nexus/auth/src/authz/oso_generic.rs:68
512025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = Rack file = nexus/auth/src/authz/oso_generic.rs:68
522025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = RoleBuiltin file = nexus/auth/src/authz/oso_generic.rs:68
532025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = SshKey file = nexus/auth/src/authz/oso_generic.rs:68
542025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = Silo file = nexus/auth/src/authz/oso_generic.rs:68
552025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = SiloUser file = nexus/auth/src/authz/oso_generic.rs:68
562025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = SiloGroup file = nexus/auth/src/authz/oso_generic.rs:68
572025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = SupportBundle file = nexus/auth/src/authz/oso_generic.rs:68
582025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = IdentityProvider file = nexus/auth/src/authz/oso_generic.rs:68
592025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = SamlIdentityProvider file = nexus/auth/src/authz/oso_generic.rs:68
602025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = Sled file = nexus/auth/src/authz/oso_generic.rs:68
612025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = TufRepo file = nexus/auth/src/authz/oso_generic.rs:68
622025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = TufArtifact file = nexus/auth/src/authz/oso_generic.rs:68
632025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = Zpool file = nexus/auth/src/authz/oso_generic.rs:68
642025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = Service file = nexus/auth/src/authz/oso_generic.rs:68
652025-03-07T22:33:50.997ZINFOnexus (ServerContext): registering Oso class class = UserBuiltin file = nexus/auth/src/authz/oso_generic.rs:68
662025-03-07T22:33:50.997ZINFOnexus (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n file = nexus/auth/src/authz/oso_generic.rs:89
672025-03-07T22:33:51.013ZINFOnexus (ServerContext): Setting up resolver using DNS servers for subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344::, width: 48 } } file = nexus/src/context.rs:222
682025-03-07T22:33:51.016ZINFOnexus (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
692025-03-07T22:33:51.020ZINFOnexus (ServerContext): Setting up qorb database pool from DNS dns_addrs = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = nexus/src/context.rs:278
702025-03-07T22:33:51.024ZDEBGnexus (ServerContext): registered USDT probes
712025-03-07T22:33:51.206ZINFOnexus: Database schema version is up to date desired_version = 129.0.0 file = nexus/db-queries/src/db/datastore/db_metadata.rs:144 found_version = 129.0.0
722025-03-07T22:33:51.210ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 789, tv_nsec: 568695454 } })
732025-03-07T22:33:51.210ZINFOnexus (SEC): SEC running file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/steno-0.4.1/src/sec.rs:813 sec_id = dfe169a3-7536-4b54-9593-8815d6e81dd4
742025-03-07T22:33:51.210ZINFOnexus (Nexus): Determining switch slots managed by switch zones file = nexus/src/app/mod.rs:1171
752025-03-07T22:33:51.260ZINFOnexus (Nexus): determining switch slot managed by dendrite zone file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
762025-03-07T22:33:51.260ZDEBGnexus (MgsClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
772025-03-07T22:33:51.265ZDEBGnexus (MgsClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "66dfc9e5-6601-405d-9ac6-c79729ced4e4", "content-length": "26", "date": "Fri, 07 Mar 2025 22:33:51 GMT"} })
782025-03-07T22:33:51.265ZINFOnexus (Nexus): identified switch slot for dendrite zone file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
792025-03-07T22:33:51.267ZINFOnexus (Nexus): completed mapping dendrite zones to switch slots file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
802025-03-07T22:33:51.298ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 789, tv_nsec: 656809471 } })
812025-03-07T22:33:51.298ZINFOnexus (Nexus): Determining switch slots managed by switch zones file = nexus/src/app/mod.rs:1171
822025-03-07T22:33:51.330ZINFOnexus (Nexus): determining switch slot managed by dendrite zone file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
832025-03-07T22:33:51.330ZDEBGnexus (MgsClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
842025-03-07T22:33:51.330ZDEBGnexus (MgsClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1d7c9db-2d4a-475b-af55-105ca40bef92", "content-length": "26", "date": "Fri, 07 Mar 2025 22:33:51 GMT"} })
852025-03-07T22:33:51.330ZINFOnexus (Nexus): identified switch slot for dendrite zone file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
862025-03-07T22:33:51.330ZINFOnexus (Nexus): completed mapping dendrite zones to switch slots file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
872025-03-07T22:33:51.395ZDEBGnexus (clickhouse-client): registered USDT probes id = 3199d66a-09e7-45b6-8db8-d23afc09152a
882025-03-07T22:33:51.406ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
892025-03-07T22:33:51.417ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
902025-03-07T22:33:51.419ZDEBGnexus (DataLoader): attempting to create built-in users
912025-03-07T22:33:51.420ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
922025-03-07T22:33:51.420ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
932025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = GET path = /bgtasks versions = all
942025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /bgtasks/activate versions = all
952025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = GET path = /bgtasks/view/{bgtask_name} versions = all
962025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = GET path = /clickhouse/policy versions = all
972025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /clickhouse/policy versions = all
982025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
992025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1002025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1012025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1022025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1032025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /demo-saga versions = all
1042025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1052025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = GET path = /deployment/blueprints/all versions = all
1062025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1072025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1082025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /deployment/blueprints/import versions = all
1092025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /deployment/blueprints/regenerate versions = all
1102025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = GET path = /deployment/blueprints/target versions = all
1112025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /deployment/blueprints/target versions = all
1122025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = PUT path = /deployment/blueprints/target/enabled versions = all
1132025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1142025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = PUT path = /disks/{disk_id} versions = all
1152025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /instances/{instance_id}/migrate versions = all
1162025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /metrics/collectors versions = all
1172025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1182025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /metrics/producers versions = all
1192025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1202025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /physical-disk/expunge versions = all
1212025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = GET path = /probes/{sled} versions = all
1222025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1232025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = GET path = /sagas versions = all
1242025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = GET path = /sagas/{saga_id} versions = all
1252025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = GET path = /sled-agents/{sled_id} versions = all
1262025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /sled-agents/{sled_id} versions = all
1272025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /sled-agents/{sled_id}/firewall-rules-update versions = all
1282025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /sleds/add versions = all
1292025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /sleds/expunge versions = all
1302025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = GET path = /sleds/uninitialized versions = all
1312025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = PUT path = /switch/{switch_id} versions = all
1322025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = PUT path = /vmms/{propolis_id} versions = all
1332025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): registered endpoint local_addr = [fd00:1122:3344:101::b]:12221 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1342025-03-07T22:33:51.434ZINFOnexus (dropshot_internal): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [fd00:1122:3344:101::b]:12221
1352025-03-07T22:33:51.434ZDEBGnexus (dropshot_internal): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::b]:12221
1362025-03-07T22:33:51.434ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
1372025-03-07T22:33:51.435ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
1382025-03-07T22:33:51.466ZWARNnexus: Cannot look up rack: Object (of type ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec)) not found: rack file = nexus/src/app/rack.rs:776
1392025-03-07T22:33:51.613ZINFOnexus (DataLoader): created 0 built-in users file = nexus/db-queries/src/db/datastore/silo_user.rs:394
1402025-03-07T22:33:51.613ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1412025-03-07T22:33:51.614ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1422025-03-07T22:33:51.614ZDEBGnexus (DataLoader): attempting to create built-in roles
1432025-03-07T22:33:51.614ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1442025-03-07T22:33:51.614ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1452025-03-07T22:33:51.631ZINFOnexus (DataLoader): created 10 built-in roles file = nexus/db-queries/src/db/datastore/role.rs:84
1462025-03-07T22:33:51.631ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1472025-03-07T22:33:51.632ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1482025-03-07T22:33:51.632ZDEBGnexus (DataLoader): attempting to create built-in role assignments
1492025-03-07T22:33:51.632ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1502025-03-07T22:33:51.633ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1512025-03-07T22:33:52.195ZINFOnexus (DataLoader): created 0 built-in role assignments file = nexus/db-queries/src/db/datastore/role.rs:113
1522025-03-07T22:33:52.195ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1532025-03-07T22:33:52.196ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1542025-03-07T22:33:52.196ZDEBGnexus (DataLoader): attempting to create built-in silos
1552025-03-07T22:33:52.196ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1562025-03-07T22:33:52.196ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1572025-03-07T22:33:52.904ZINFOnexus (DataLoader): created 0 built-in silos file = nexus/db-queries/src/db/datastore/silo.rs:93
1582025-03-07T22:33:52.905ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1592025-03-07T22:33:52.905ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1602025-03-07T22:33:52.919ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1612025-03-07T22:33:52.920ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1622025-03-07T22:33:52.920ZDEBGnexus (DataLoader): attempting to create built-in projects
1632025-03-07T22:33:52.920ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1642025-03-07T22:33:52.920ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1652025-03-07T22:33:52.965ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1662025-03-07T22:33:52.965ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1672025-03-07T22:33:52.968ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1682025-03-07T22:33:52.969ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1692025-03-07T22:33:53.001ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1702025-03-07T22:33:53.004ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
1712025-03-07T22:33:53.004ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1722025-03-07T22:33:53.004ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1732025-03-07T22:33:53.040ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1742025-03-07T22:33:53.040ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1752025-03-07T22:33:53.093ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1762025-03-07T22:33:53.094ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
1772025-03-07T22:33:53.094ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1782025-03-07T22:33:53.094ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1792025-03-07T22:33:53.148ZINFOnexus (DataLoader): created built-in services project file = nexus/db-queries/src/db/datastore/project.rs:122
1802025-03-07T22:33:53.148ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1812025-03-07T22:33:53.149ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1822025-03-07T22:33:53.149ZDEBGnexus (DataLoader): attempting to create built-in VPCs
1832025-03-07T22:33:53.149ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1842025-03-07T22:33:53.149ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1852025-03-07T22:33:53.152ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1862025-03-07T22:33:53.152ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1872025-03-07T22:33:53.157ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1882025-03-07T22:33:53.157ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1892025-03-07T22:33:53.158ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1902025-03-07T22:33:53.158ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1912025-03-07T22:33:53.160ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1922025-03-07T22:33:53.160ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1932025-03-07T22:33:53.201ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1942025-03-07T22:33:53.204ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
1952025-03-07T22:33:53.205ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1962025-03-07T22:33:53.205ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1972025-03-07T22:33:53.206ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
1982025-03-07T22:33:53.207ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1992025-03-07T22:33:53.208ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2002025-03-07T22:33:53.209ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2012025-03-07T22:33:53.210ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2022025-03-07T22:33:53.212ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2032025-03-07T22:33:53.212ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2042025-03-07T22:33:53.213ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2052025-03-07T22:33:53.312ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2062025-03-07T22:33:53.312ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2072025-03-07T22:33:53.316ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2082025-03-07T22:33:53.316ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2092025-03-07T22:33:53.318ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2102025-03-07T22:33:53.318ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2112025-03-07T22:33:53.320ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2122025-03-07T22:33:53.323ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
2132025-03-07T22:33:53.323ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2142025-03-07T22:33:53.323ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2152025-03-07T22:33:53.346ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2162025-03-07T22:33:53.346ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2172025-03-07T22:33:53.357ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2182025-03-07T22:33:53.358ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2192025-03-07T22:33:53.371ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2202025-03-07T22:33:53.371ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2212025-03-07T22:33:53.373ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2222025-03-07T22:33:53.373ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2232025-03-07T22:33:53.374ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2242025-03-07T22:33:53.375ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2252025-03-07T22:33:53.376ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2262025-03-07T22:33:53.376ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2272025-03-07T22:33:53.377ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2282025-03-07T22:33:53.378ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2292025-03-07T22:33:53.379ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2302025-03-07T22:33:53.382ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
2312025-03-07T22:33:53.382ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2322025-03-07T22:33:53.383ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2332025-03-07T22:33:53.384ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2342025-03-07T22:33:53.384ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2352025-03-07T22:33:53.386ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2362025-03-07T22:33:53.386ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2372025-03-07T22:33:53.387ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2382025-03-07T22:33:53.391ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
2392025-03-07T22:33:53.391ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2402025-03-07T22:33:53.391ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2412025-03-07T22:33:53.439ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2422025-03-07T22:33:53.440ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2432025-03-07T22:33:53.441ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2442025-03-07T22:33:53.441ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2452025-03-07T22:33:53.442ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2462025-03-07T22:33:53.443ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2472025-03-07T22:33:53.444ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2482025-03-07T22:33:53.448ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
2492025-03-07T22:33:53.448ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2502025-03-07T22:33:53.448ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2512025-03-07T22:33:53.467ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
2522025-03-07T22:33:53.467ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
2532025-03-07T22:33:53.496ZWARNnexus: Cannot look up rack: Object (of type ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec)) not found: rack file = nexus/src/app/rack.rs:776
2542025-03-07T22:33:53.552ZDEBGnexus (DataLoader): attempting to create built-in VPC firewall rules
2552025-03-07T22:33:53.552ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2562025-03-07T22:33:53.553ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2572025-03-07T22:33:53.558ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2582025-03-07T22:33:53.558ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2592025-03-07T22:33:53.560ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2602025-03-07T22:33:53.560ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2612025-03-07T22:33:53.562ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2622025-03-07T22:33:53.563ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2632025-03-07T22:33:53.564ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2642025-03-07T22:33:53.564ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2652025-03-07T22:33:53.565ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2662025-03-07T22:33:53.566ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2672025-03-07T22:33:53.567ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2682025-03-07T22:33:53.572ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
2692025-03-07T22:33:53.572ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2702025-03-07T22:33:53.572ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2712025-03-07T22:33:53.574ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2722025-03-07T22:33:53.574ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2732025-03-07T22:33:53.575ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2742025-03-07T22:33:53.575ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2752025-03-07T22:33:53.577ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2762025-03-07T22:33:53.580ZDEBGnexus (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
2772025-03-07T22:33:53.580ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2782025-03-07T22:33:53.581ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2792025-03-07T22:33:53.606ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2802025-03-07T22:33:53.606ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2812025-03-07T22:33:53.607ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2822025-03-07T22:33:53.607ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2832025-03-07T22:33:53.609ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2842025-03-07T22:33:53.609ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2852025-03-07T22:33:53.610ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2862025-03-07T22:33:53.613ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
2872025-03-07T22:33:53.613ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2882025-03-07T22:33:53.614ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2892025-03-07T22:33:53.702ZINFOnexus (DataLoader): created built-in VPC firewall rules file = nexus/db-queries/src/db/datastore/vpc.rs:284
2902025-03-07T22:33:53.702ZDEBGnexus (DataLoader): attempting to create built-in VPC Subnets
2912025-03-07T22:33:53.702ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2922025-03-07T22:33:53.702ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2932025-03-07T22:33:53.713ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2942025-03-07T22:33:53.713ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2952025-03-07T22:33:53.715ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2962025-03-07T22:33:53.715ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2972025-03-07T22:33:53.717ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
2982025-03-07T22:33:53.717ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2992025-03-07T22:33:53.719ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3002025-03-07T22:33:53.719ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3012025-03-07T22:33:53.720ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3022025-03-07T22:33:53.720ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3032025-03-07T22:33:53.722ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3042025-03-07T22:33:53.725ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3052025-03-07T22:33:53.725ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3062025-03-07T22:33:53.725ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3072025-03-07T22:33:53.732ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3082025-03-07T22:33:53.732ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3092025-03-07T22:33:53.734ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3102025-03-07T22:33:53.734ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3112025-03-07T22:33:53.736ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3122025-03-07T22:33:53.736ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3132025-03-07T22:33:53.749ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3142025-03-07T22:33:53.749ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3152025-03-07T22:33:53.752ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3162025-03-07T22:33:53.753ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3172025-03-07T22:33:53.754ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3182025-03-07T22:33:53.754ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3192025-03-07T22:33:53.756ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3202025-03-07T22:33:53.761ZDEBGnexus (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ById(001de000-c470-4000-8000-000000000001) } result = Ok(())
3212025-03-07T22:33:53.761ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3222025-03-07T22:33:53.761ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3232025-03-07T22:33:53.763ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3242025-03-07T22:33:53.763ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3252025-03-07T22:33:53.765ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3262025-03-07T22:33:53.765ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3272025-03-07T22:33:53.767ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3282025-03-07T22:33:53.767ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3292025-03-07T22:33:53.768ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3302025-03-07T22:33:53.769ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3312025-03-07T22:33:53.770ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3322025-03-07T22:33:53.770ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3332025-03-07T22:33:53.771ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3342025-03-07T22:33:53.771ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3352025-03-07T22:33:53.772ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3362025-03-07T22:33:53.776ZDEBGnexus (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ById(001de000-c470-4000-8000-000000000002) } result = Ok(())
3372025-03-07T22:33:53.776ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3382025-03-07T22:33:53.776ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3392025-03-07T22:33:53.778ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3402025-03-07T22:33:53.778ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3412025-03-07T22:33:53.780ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3422025-03-07T22:33:53.780ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3432025-03-07T22:33:53.781ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3442025-03-07T22:33:53.781ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3452025-03-07T22:33:53.783ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3462025-03-07T22:33:53.786ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3472025-03-07T22:33:53.798ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3482025-03-07T22:33:53.798ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3492025-03-07T22:33:53.800ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3502025-03-07T22:33:53.800ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3512025-03-07T22:33:53.801ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3522025-03-07T22:33:53.802ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3532025-03-07T22:33:53.803ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3542025-03-07T22:33:53.806ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3552025-03-07T22:33:53.806ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3562025-03-07T22:33:53.806ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3572025-03-07T22:33:53.893ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3582025-03-07T22:33:53.893ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3592025-03-07T22:33:53.897ZINFOnexus (DataLoader): created built-in services vpc subnets file = nexus/db-queries/src/db/datastore/vpc.rs:350
3602025-03-07T22:33:53.897ZINFOnexus (DataLoader): created built-in services vpc file = nexus/db-queries/src/db/datastore/vpc.rs:226
3612025-03-07T22:33:53.897ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3622025-03-07T22:33:53.898ZDEBGnexus (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3632025-03-07T22:33:53.898ZDEBGnexus (DataLoader): attempting to create silo users
3642025-03-07T22:33:53.898ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3652025-03-07T22:33:53.898ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3662025-03-07T22:33:53.912ZINFOnexus (DataLoader): created 0 silo users file = nexus/db-queries/src/db/datastore/silo_user.rs:419
3672025-03-07T22:33:53.912ZDEBGnexus (DataLoader): attempting to create silo user role assignments
3682025-03-07T22:33:53.912ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3692025-03-07T22:33:53.912ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3702025-03-07T22:33:53.916ZINFOnexus (DataLoader): created 0 silo user role assignments file = nexus/db-queries/src/db/datastore/silo_user.rs:446
3712025-03-07T22:33:53.916ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3722025-03-07T22:33:53.916ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3732025-03-07T22:33:53.918ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3742025-03-07T22:33:53.918ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3752025-03-07T22:33:53.923ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3762025-03-07T22:33:53.923ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3772025-03-07T22:33:53.925ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3782025-03-07T22:33:53.926ZDEBGnexus (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
3792025-03-07T22:33:53.926ZDEBGnexus (DataLoader): roles roles = RoleSet { roles: {} }
3802025-03-07T22:33:53.926ZDEBGnexus (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3812025-03-07T22:33:53.939ZINFOnexus: populate complete file = nexus/src/app/mod.rs:545
3822025-03-07T22:33:53.939ZINFOnexus: activating background tasks file = nexus/src/app/mod.rs:554
3832025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
3842025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
3852025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
3862025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
3872025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
3882025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
3892025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
3902025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
3912025-03-07T22:33:53.959ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector file = nexus/src/app/background/tasks/crdb_node_id_collector.rs:78 reason = no blueprint
3922025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
3932025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
3942025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
3952025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
3962025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
3972025-03-07T22:33:53.959ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:61 reason = no config nor servers
3982025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
3992025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
4002025-03-07T22:33:53.959ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:71 reason = no blueprint
4012025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
4022025-03-07T22:33:53.959ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
4032025-03-07T22:33:53.959ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
4042025-03-07T22:33:53.960ZWARNnexus (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous file = nexus/src/app/background/tasks/blueprint_rendezvous.rs:52 reason = no blueprint
4052025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4062025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
4072025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
4082025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
4092025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
4102025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
4112025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
4122025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
4132025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
4142025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
4152025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
4162025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
4172025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
4182025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
4192025-03-07T22:33:53.960ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:23:53.959960659 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
4202025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 340.434\u{b5}s iteration = 1 status = {"error":"no blueprint"}
4212025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 257.843\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
4222025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
4232025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
4242025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
4252025-03-07T22:33:53.960ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
4262025-03-07T22:33:53.961ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4272025-03-07T22:33:53.961ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
4282025-03-07T22:33:53.961ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 1 reason = Timeout
4292025-03-07T22:33:53.961ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:61 reason = no config nor servers
4302025-03-07T22:33:53.961ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
4312025-03-07T22:33:53.961ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 1.853348ms iteration = 1 status = {"error":"no config nor servers"}
4322025-03-07T22:33:53.961ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
4332025-03-07T22:33:53.961ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
4342025-03-07T22:33:53.961ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
4352025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4362025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4372025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
4382025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
4392025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4402025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
4412025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4422025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
4432025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
4442025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
4452025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
4462025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
4472025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 362.184\u{b5}s iteration = 1 status = {"error":"no blueprint"}
4482025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
4492025-03-07T22:33:53.962ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
4502025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 402.524\u{b5}s iteration = 1 status = {"error":"no blueprint"}
4512025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
4522025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
4532025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
4542025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
4552025-03-07T22:33:53.962ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
4562025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
4572025-03-07T22:33:53.962ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
4582025-03-07T22:33:53.963ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
4592025-03-07T22:33:53.964ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 792, tv_nsec: 321458378 } })
4602025-03-07T22:33:53.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4612025-03-07T22:33:53.964ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
4622025-03-07T22:33:53.964ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
4632025-03-07T22:33:53.964ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4642025-03-07T22:33:53.964ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 792, tv_nsec: 323050263 } })
4652025-03-07T22:33:53.964ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
4662025-03-07T22:33:53.964ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
4672025-03-07T22:33:53.964ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
4682025-03-07T22:33:53.965ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
4692025-03-07T22:33:53.965ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4702025-03-07T22:33:53.965ZINFOnexus (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external dns_group = external file = nexus/src/app/background/tasks/dns_servers.rs:113
4712025-03-07T22:33:53.965ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 5.436802ms iteration = 1 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
4722025-03-07T22:33:53.965ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
4732025-03-07T22:33:53.965ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:69 reason = no config
4742025-03-07T22:33:53.965ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 8.75\u{b5}s iteration = 2 status = {"error":"no config"}
4752025-03-07T22:33:53.965ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4762025-03-07T22:33:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
4772025-03-07T22:33:53.965ZINFOnexus (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal dns_group = internal file = nexus/src/app/background/tasks/dns_servers.rs:113
4782025-03-07T22:33:53.965ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 5.730875ms iteration = 1 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
4792025-03-07T22:33:53.965ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
4802025-03-07T22:33:53.965ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:69 reason = no config
4812025-03-07T22:33:53.965ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 3.09\u{b5}s iteration = 2 status = {"error":"no config"}
4822025-03-07T22:33:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4832025-03-07T22:33:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
4842025-03-07T22:33:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
4852025-03-07T22:33:53.967ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4862025-03-07T22:33:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4872025-03-07T22:33:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4882025-03-07T22:33:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4892025-03-07T22:33:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4902025-03-07T22:33:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
4912025-03-07T22:33:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
4922025-03-07T22:33:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
4932025-03-07T22:33:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
4942025-03-07T22:33:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
4952025-03-07T22:33:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4962025-03-07T22:33:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
4972025-03-07T22:33:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
4982025-03-07T22:33:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
4992025-03-07T22:33:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5002025-03-07T22:33:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
5012025-03-07T22:33:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
5022025-03-07T22:33:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5032025-03-07T22:33:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5042025-03-07T22:33:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
5052025-03-07T22:33:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5062025-03-07T22:33:53.969ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
5072025-03-07T22:33:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
5082025-03-07T22:33:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
5092025-03-07T22:33:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
5102025-03-07T22:33:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5112025-03-07T22:33:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5122025-03-07T22:33:53.970ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
5132025-03-07T22:33:53.970ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5142025-03-07T22:33:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5152025-03-07T22:33:53.971ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5162025-03-07T22:33:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5172025-03-07T22:33:53.972ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5182025-03-07T22:33:53.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5192025-03-07T22:33:53.975ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5202025-03-07T22:33:53.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5212025-03-07T22:33:53.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 792, tv_nsec: 344662751 } })
5222025-03-07T22:33:53.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
5232025-03-07T22:33:53.990ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 30.522644ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
5242025-03-07T22:33:53.996ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5252025-03-07T22:33:53.997ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5262025-03-07T22:33:54.000ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
5272025-03-07T22:33:54.000ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
5282025-03-07T22:33:54.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5292025-03-07T22:33:54.000ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5302025-03-07T22:33:54.000ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5312025-03-07T22:33:54.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5322025-03-07T22:33:54.003ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 792, tv_nsec: 361476792 } })
5332025-03-07T22:33:54.003ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
5342025-03-07T22:33:54.007ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 47.016322ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
5352025-03-07T22:33:54.036ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5362025-03-07T22:33:54.037ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5372025-03-07T22:33:54.038ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
5382025-03-07T22:33:54.039ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5392025-03-07T22:33:54.039ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5402025-03-07T22:33:54.039ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
5412025-03-07T22:33:54.039ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5422025-03-07T22:33:54.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5432025-03-07T22:33:54.040ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "92e6ce21-2387-43f6-87bb-99781bef0177", "content-length": "26", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
5442025-03-07T22:33:54.042ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5452025-03-07T22:33:54.042ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5462025-03-07T22:33:54.042ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5472025-03-07T22:33:54.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5482025-03-07T22:33:54.042ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
5492025-03-07T22:33:54.042ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
5502025-03-07T22:33:54.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
5512025-03-07T22:33:54.044ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
5522025-03-07T22:33:54.051ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5532025-03-07T22:33:54.051ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5542025-03-07T22:33:54.051ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5552025-03-07T22:33:54.055ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
5562025-03-07T22:33:54.055ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
5572025-03-07T22:33:54.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
5582025-03-07T22:33:54.055ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
5592025-03-07T22:33:54.055ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5602025-03-07T22:33:54.055ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5612025-03-07T22:33:54.055ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5622025-03-07T22:33:54.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5632025-03-07T22:33:54.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5642025-03-07T22:33:54.058ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:23:53.959960659 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
5652025-03-07T22:33:54.058ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5662025-03-07T22:33:54.058ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 98.775859ms iteration = 1 status = {"expiration":"2025-03-07T22:23:53.959960659Z","pruned":[]}
5672025-03-07T22:33:54.061ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5682025-03-07T22:33:54.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5692025-03-07T22:33:54.061ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
5702025-03-07T22:33:54.061ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5712025-03-07T22:33:54.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5722025-03-07T22:33:54.062ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
5732025-03-07T22:33:54.062ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5742025-03-07T22:33:54.062ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "60e1e3d2-2404-4c1f-8ebe-9ad72bb7be97", "content-length": "26", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
5752025-03-07T22:33:54.062ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5762025-03-07T22:33:54.064ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5772025-03-07T22:33:54.064ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
5782025-03-07T22:33:54.064ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5792025-03-07T22:33:54.065ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5802025-03-07T22:33:54.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5812025-03-07T22:33:54.065ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5822025-03-07T22:33:54.067ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5832025-03-07T22:33:54.067ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
5842025-03-07T22:33:54.067ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5852025-03-07T22:33:54.067ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
5862025-03-07T22:33:54.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5872025-03-07T22:33:54.069ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5882025-03-07T22:33:54.069ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
5892025-03-07T22:33:54.069ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
5902025-03-07T22:33:54.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
5912025-03-07T22:33:54.072ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5922025-03-07T22:33:54.074ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
5932025-03-07T22:33:54.074ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5942025-03-07T22:33:54.075ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 114.982325ms iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
5952025-03-07T22:33:54.075ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5962025-03-07T22:33:54.075ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5972025-03-07T22:33:54.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5982025-03-07T22:33:54.075ZINFOnexus (BackgroundTasks): Physical Disk Adoption: Rack not yet initialized background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:80 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
5992025-03-07T22:33:54.075ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 115.443949ms iteration = 1 status = {"error":"rack not yet initialized: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec"}
6002025-03-07T22:33:54.076ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
6012025-03-07T22:33:54.076ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
6022025-03-07T22:33:54.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
6032025-03-07T22:33:54.079ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 119.310607ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
6042025-03-07T22:33:54.080ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6052025-03-07T22:33:54.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6062025-03-07T22:33:54.083ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6072025-03-07T22:33:54.085ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
6082025-03-07T22:33:54.085ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
6092025-03-07T22:33:54.085ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 125.728978ms iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
6102025-03-07T22:33:54.085ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 124.981131ms iteration = 1 status = {"errors":[],"requests_created_ok":[]}
6112025-03-07T22:33:54.090ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
6122025-03-07T22:33:54.090ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6132025-03-07T22:33:54.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6142025-03-07T22:33:54.101ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 142.020955ms iteration = 1 status = {}
6152025-03-07T22:33:54.104ZWARNnexus (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set file = nexus/src/app/background/tasks/blueprint_load.rs:69
6162025-03-07T22:33:54.104ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 144.036994ms iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
6172025-03-07T22:33:54.104ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
6182025-03-07T22:33:54.104ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
6192025-03-07T22:33:54.106ZWARNnexus (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker file = nexus/src/app/background/tasks/sync_service_zone_nat.rs:95
6202025-03-07T22:33:54.106ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 146.012593ms iteration = 1 status = {"error":"inventory collection is None"}
6212025-03-07T22:33:54.106ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 145.73777ms iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
6222025-03-07T22:33:54.106ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
6232025-03-07T22:33:54.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
6242025-03-07T22:33:54.117ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6252025-03-07T22:33:54.117ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
6262025-03-07T22:33:54.117ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
6272025-03-07T22:33:54.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
6282025-03-07T22:33:54.138ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
6292025-03-07T22:33:54.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
6302025-03-07T22:33:54.143ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6312025-03-07T22:33:54.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6322025-03-07T22:33:54.144ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
6332025-03-07T22:33:54.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
6342025-03-07T22:33:54.145ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
6352025-03-07T22:33:54.145ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 792, tv_nsec: 504355915 } })
6362025-03-07T22:33:54.151ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6372025-03-07T22:33:54.154ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
6382025-03-07T22:33:54.155ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6392025-03-07T22:33:54.155ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6402025-03-07T22:33:54.164ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
6412025-03-07T22:33:54.164ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
6422025-03-07T22:33:54.165ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 205.173182ms iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
6432025-03-07T22:33:54.167ZWARNnexus (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0 file = nexus/src/app/background/tasks/dns_config.rs:72
6442025-03-07T22:33:54.167ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 207.761127ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
6452025-03-07T22:33:54.175ZWARNnexus (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0 file = nexus/src/app/background/tasks/dns_config.rs:72
6462025-03-07T22:33:54.175ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 215.913455ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
6472025-03-07T22:33:54.182ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6482025-03-07T22:33:54.183ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6492025-03-07T22:33:54.184ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 224.214775ms iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
6502025-03-07T22:33:54.185ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6512025-03-07T22:33:54.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6522025-03-07T22:33:54.187ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6532025-03-07T22:33:54.193ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
6542025-03-07T22:33:54.193ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6552025-03-07T22:33:54.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6562025-03-07T22:33:54.196ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6572025-03-07T22:33:54.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6582025-03-07T22:33:54.197ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
6592025-03-07T22:33:54.197ZWARNnexus (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone file = nexus/src/app/background/tasks/external_endpoints.rs:48
6602025-03-07T22:33:54.197ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 237.630224ms iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
6612025-03-07T22:33:54.199ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6622025-03-07T22:33:54.200ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6632025-03-07T22:33:54.210ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "62a5b547-6147-4043-9443-a0f48562fff7", "content-length": "1", "date": "Fri, 07 Mar 2025 22:33:53 GMT"} })
6642025-03-07T22:33:54.212ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6652025-03-07T22:33:54.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6662025-03-07T22:33:54.214ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
6672025-03-07T22:33:54.214ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 254.655547ms iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
6682025-03-07T22:33:54.222ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6692025-03-07T22:33:54.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6702025-03-07T22:33:54.223ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 263.776934ms iteration = 1 status = {}
6712025-03-07T22:33:54.224ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 264.165429ms iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
6722025-03-07T22:33:54.226ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
6732025-03-07T22:33:54.226ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
6742025-03-07T22:33:54.226ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
6752025-03-07T22:33:54.227ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6eaaf777-ef87-41b5-89f3-32edfa5b59df", "content-length": "476", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
6762025-03-07T22:33:54.228ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 268.416339ms iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6772025-03-07T22:33:54.229ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 269.325508ms iteration = 1 status = null
6782025-03-07T22:33:54.231ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
6792025-03-07T22:33:54.231ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bde8d815-9520-4d56-9a13-1fa5fb5f2ea9", "content-length": "734", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
6802025-03-07T22:33:54.232ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
6812025-03-07T22:33:54.233ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "182ab4b1-eb29-4c69-aea4-b9be0dfec4d1", "content-length": "111", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
6822025-03-07T22:33:54.234ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
6832025-03-07T22:33:54.234ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6842025-03-07T22:33:54.234ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
6852025-03-07T22:33:54.234ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
6862025-03-07T22:33:54.234ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e201b05f-fb07-43bf-968d-05008a280132", "content-length": "111", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
6872025-03-07T22:33:54.234ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
6882025-03-07T22:33:54.234ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
6892025-03-07T22:33:54.235ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02877620-3231-4e48-a2a5-8180db0cbd11", "content-length": "167", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
6902025-03-07T22:33:54.235ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
6912025-03-07T22:33:54.235ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1338eab2-7fd8-49e2-b829-8a0ce20efc22", "content-length": "167", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
6922025-03-07T22:33:54.235ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
6932025-03-07T22:33:54.236ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbb1a476-c3cc-4d15-a572-242baf0fcae7", "content-length": "175", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
6942025-03-07T22:33:54.236ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
6952025-03-07T22:33:54.236ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 276.087253ms iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6962025-03-07T22:33:54.236ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ce5a6c2-b696-4988-8ef8-2f5b367d7806", "content-length": "175", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
6972025-03-07T22:33:54.236ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
6982025-03-07T22:33:54.238ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fa081b2-d79a-4500-8948-4aa11fc9aaa1", "content-length": "702", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
6992025-03-07T22:33:54.238ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
7002025-03-07T22:33:54.248ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1a90604-9c1f-4097-b675-e15e11b7a3a1", "content-length": "727", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7012025-03-07T22:33:54.249ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
7022025-03-07T22:33:54.250ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 290.126439ms iteration = 1 status = {}
7032025-03-07T22:33:54.250ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0b26853-a88d-46a4-9c2e-466a9b3835ea", "content-length": "729", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7042025-03-07T22:33:54.250ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
7052025-03-07T22:33:54.250ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "badc4e7c-2c1d-42af-ade0-16cced5a7e7c", "content-length": "728", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7062025-03-07T22:33:54.250ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
7072025-03-07T22:33:54.251ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc9afb2c-8046-4d40-97b9-576e29d26359", "content-length": "722", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7082025-03-07T22:33:54.251ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
7092025-03-07T22:33:54.251ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a1aeb7a-dd21-4e06-9da0-081ee72781d5", "content-length": "109", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7102025-03-07T22:33:54.251ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
7112025-03-07T22:33:54.252ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61b077a9-6892-4c72-ba73-72ef906bf5c5", "content-length": "109", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7122025-03-07T22:33:54.252ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
7132025-03-07T22:33:54.252ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e4e29c4-d7b0-4581-9d90-88ba496becf1", "content-length": "169", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7142025-03-07T22:33:54.252ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
7152025-03-07T22:33:54.252ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ee1453d-d806-4e56-8cef-3049f90e3fec", "content-length": "169", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7162025-03-07T22:33:54.252ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
7172025-03-07T22:33:54.253ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c95ad16-54be-4408-a449-af9c03bab3df", "content-length": "178", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7182025-03-07T22:33:54.253ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
7192025-03-07T22:33:54.253ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da461b61-a4fd-4218-9fca-5dcd84f1e1a2", "content-length": "178", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7202025-03-07T22:33:54.253ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
7212025-03-07T22:33:54.253ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aba7e668-444e-48f9-bfc7-f9aa9e61b3de", "content-length": "702", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7222025-03-07T22:33:54.253ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
7232025-03-07T22:33:54.254ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eeb19de6-0268-453e-8292-7ee484da0b6c", "content-length": "727", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7242025-03-07T22:33:54.254ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
7252025-03-07T22:33:54.254ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed6a6db2-42d8-498f-bd06-dbea0e87a59d", "content-length": "729", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7262025-03-07T22:33:54.254ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
7272025-03-07T22:33:54.254ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "013a7175-0d57-4dbb-b887-5f0bb23d68ca", "content-length": "728", "date": "Fri, 07 Mar 2025 22:33:54 GMT"} })
7282025-03-07T22:33:54.254ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
7292025-03-07T22:33:54.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
7302025-03-07T22:33:54.256ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7312025-03-07T22:33:54.259ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
7322025-03-07T22:33:54.259ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
7332025-03-07T22:33:54.259ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
7342025-03-07T22:33:54.261ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7352025-03-07T22:33:54.263ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
7362025-03-07T22:33:54.263ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 303.276034ms iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
7372025-03-07T22:33:54.265ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
7382025-03-07T22:33:54.265ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7392025-03-07T22:33:54.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7402025-03-07T22:33:54.273ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7412025-03-07T22:33:54.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7422025-03-07T22:33:54.275ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7432025-03-07T22:33:54.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7442025-03-07T22:33:54.277ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
7452025-03-07T22:33:54.277ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7462025-03-07T22:33:54.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
7472025-03-07T22:33:54.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7482025-03-07T22:33:54.286ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7492025-03-07T22:33:54.288ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7502025-03-07T22:33:54.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7512025-03-07T22:33:54.289ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
7522025-03-07T22:33:54.289ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
7532025-03-07T22:33:54.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
7542025-03-07T22:33:54.291ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7552025-03-07T22:33:54.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7562025-03-07T22:33:54.293ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7572025-03-07T22:33:54.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7582025-03-07T22:33:54.295ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7592025-03-07T22:33:54.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
7602025-03-07T22:33:54.302ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7612025-03-07T22:33:54.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7622025-03-07T22:33:54.306ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7632025-03-07T22:33:54.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7642025-03-07T22:33:54.309ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7652025-03-07T22:33:54.309ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 349.298047ms iteration = 1 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":0,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":0,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[]}
7662025-03-07T22:33:54.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7672025-03-07T22:33:54.312ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7682025-03-07T22:33:54.312ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7692025-03-07T22:33:54.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7702025-03-07T22:33:54.313ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
7712025-03-07T22:33:54.313ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
7722025-03-07T22:33:54.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
7732025-03-07T22:33:54.315ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7742025-03-07T22:33:54.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7752025-03-07T22:33:54.317ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7762025-03-07T22:33:54.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7772025-03-07T22:33:54.319ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7782025-03-07T22:33:54.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7792025-03-07T22:33:54.320ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7802025-03-07T22:33:54.323ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
7812025-03-07T22:33:54.323ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
7822025-03-07T22:33:54.323ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 363.725845ms iteration = 1 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7832025-03-07T22:33:54.325ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
7842025-03-07T22:33:54.325ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7852025-03-07T22:33:54.325ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7862025-03-07T22:33:54.326ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7872025-03-07T22:33:54.327ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7882025-03-07T22:33:54.328ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7892025-03-07T22:33:54.329ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7902025-03-07T22:33:54.330ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7912025-03-07T22:33:54.334ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
7922025-03-07T22:33:54.334ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7932025-03-07T22:33:54.335ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7942025-03-07T22:33:54.340ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
7952025-03-07T22:33:54.340ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 381.002611ms iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
7962025-03-07T22:33:54.353ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7972025-03-07T22:33:54.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7982025-03-07T22:33:54.355ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7992025-03-07T22:33:54.355ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8002025-03-07T22:33:54.356ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8012025-03-07T22:33:54.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8022025-03-07T22:33:54.361ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8032025-03-07T22:33:54.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8042025-03-07T22:33:54.364ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8052025-03-07T22:33:54.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8062025-03-07T22:33:54.366ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8072025-03-07T22:33:54.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8082025-03-07T22:33:54.368ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8092025-03-07T22:33:54.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8102025-03-07T22:33:54.369ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8112025-03-07T22:33:54.375ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
8122025-03-07T22:33:54.375ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8132025-03-07T22:33:54.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8142025-03-07T22:33:54.377ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8152025-03-07T22:33:54.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8162025-03-07T22:33:54.380ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8172025-03-07T22:33:54.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8182025-03-07T22:33:54.380ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
8192025-03-07T22:33:54.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8202025-03-07T22:33:54.382ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8212025-03-07T22:33:54.383ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8222025-03-07T22:33:54.386ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
8232025-03-07T22:33:54.386ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
8242025-03-07T22:33:54.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8252025-03-07T22:33:54.386ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
8262025-03-07T22:33:54.386ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8272025-03-07T22:33:54.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8282025-03-07T22:33:54.387ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8292025-03-07T22:33:54.389ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
8302025-03-07T22:33:54.389ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
8312025-03-07T22:33:54.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8322025-03-07T22:33:54.402ZDEBGnexus (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
8332025-03-07T22:33:54.402ZDEBGnexus (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
8342025-03-07T22:33:54.402ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
8352025-03-07T22:33:54.403ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
8362025-03-07T22:33:54.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
8372025-03-07T22:33:54.404ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 444.703324ms iteration = 1 status = {}
8382025-03-07T22:33:54.404ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
8392025-03-07T22:33:54.405ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8402025-03-07T22:33:54.405ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8412025-03-07T22:33:54.405ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8422025-03-07T22:33:54.407ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8432025-03-07T22:33:54.408ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
8442025-03-07T22:33:54.408ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
8452025-03-07T22:33:54.408ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
8462025-03-07T22:33:54.409ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
8472025-03-07T22:33:54.409ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8482025-03-07T22:33:54.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8492025-03-07T22:33:54.422ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
8502025-03-07T22:33:54.422ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
8512025-03-07T22:33:54.422ZDEBGnexus (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
8522025-03-07T22:33:54.541ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
8532025-03-07T22:33:54.541ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
8542025-03-07T22:33:54.541ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
8552025-03-07T22:33:54.542ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 579.145446ms
8562025-03-07T22:33:54.542ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 581.74116ms iteration = 1 status = {}
8572025-03-07T22:33:54.905ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = e40db5c6-aed6-4e29-9fe0-e1223dbce78b file = nexus/db-queries/src/db/datastore/inventory.rs:1010
8582025-03-07T22:33:54.905ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = e40db5c6-aed6-4e29-9fe0-e1223dbce78b time_started = 2025-03-07 22:33:54.225546 UTC
8592025-03-07T22:33:54.905ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 945.572167ms iteration = 1 status = {"collection_id":"e40db5c6-aed6-4e29-9fe0-e1223dbce78b","time_done":"2025-03-07 22:33:54.402246 UTC","time_started":"2025-03-07 22:33:54.225546 UTC"}
8602025-03-07T22:33:54.905ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
8612025-03-07T22:33:54.905ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
8622025-03-07T22:33:54.905ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8632025-03-07T22:33:54.905ZWARNnexus (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous file = nexus/src/app/background/tasks/blueprint_rendezvous.rs:52 reason = no blueprint
8642025-03-07T22:33:54.905ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 30.821\u{b5}s iteration = 2 status = {"error":"no blueprint"}
8652025-03-07T22:33:54.905ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8662025-03-07T22:33:54.907ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8672025-03-07T22:33:54.909ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
8682025-03-07T22:33:54.909ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8692025-03-07T22:33:54.909ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8702025-03-07T22:33:54.910ZINFOnexus (BackgroundTasks): Physical Disk Adoption: Rack not yet initialized background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:80 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
8712025-03-07T22:33:54.910ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 5.594134ms iteration = 2 status = {"error":"rack not yet initialized: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec"}
8722025-03-07T22:33:55.497ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8732025-03-07T22:33:55.497ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8742025-03-07T22:33:55.499ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8752025-03-07T22:33:55.499ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8762025-03-07T22:33:55.501ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8772025-03-07T22:33:55.503ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, lookup_type: ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec) } result = Ok(())
8782025-03-07T22:33:55.503ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, time_created: 2025-03-07T22:33:53.849453Z, time_modified: 2025-03-07T22:33:53.849453Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
8792025-03-07T22:33:57.504ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8802025-03-07T22:33:57.505ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8812025-03-07T22:33:57.508ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8822025-03-07T22:33:57.508ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8832025-03-07T22:33:57.510ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8842025-03-07T22:33:57.512ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, lookup_type: ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec) } result = Ok(())
8852025-03-07T22:33:57.512ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, time_created: 2025-03-07T22:33:53.849453Z, time_modified: 2025-03-07T22:33:53.849453Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
8862025-03-07T22:33:59.513ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8872025-03-07T22:33:59.513ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8882025-03-07T22:33:59.514ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8892025-03-07T22:33:59.515ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8902025-03-07T22:33:59.517ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8912025-03-07T22:33:59.519ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, lookup_type: ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec) } result = Ok(())
8922025-03-07T22:33:59.519ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, time_created: 2025-03-07T22:33:53.849453Z, time_modified: 2025-03-07T22:33:53.849453Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
8932025-03-07T22:34:01.521ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8942025-03-07T22:34:01.521ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8952025-03-07T22:34:01.523ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
8962025-03-07T22:34:01.523ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
8972025-03-07T22:34:01.525ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8982025-03-07T22:34:01.527ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, lookup_type: ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec) } result = Ok(())
8992025-03-07T22:34:01.527ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, time_created: 2025-03-07T22:33:53.849453Z, time_modified: 2025-03-07T22:33:53.849453Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
9002025-03-07T22:34:03.529ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9012025-03-07T22:34:03.529ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9022025-03-07T22:34:03.531ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9032025-03-07T22:34:03.531ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9042025-03-07T22:34:03.532ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9052025-03-07T22:34:03.535ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, lookup_type: ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec) } result = Ok(())
9062025-03-07T22:34:03.535ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, time_created: 2025-03-07T22:33:53.849453Z, time_modified: 2025-03-07T22:33:53.849453Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
9072025-03-07T22:34:03.959ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 2 reason = Timeout
9082025-03-07T22:34:03.959ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 2 reason = Timeout
9092025-03-07T22:34:03.959ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9102025-03-07T22:34:03.959ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9112025-03-07T22:34:03.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9122025-03-07T22:34:03.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9132025-03-07T22:34:03.962ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9142025-03-07T22:34:03.964ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9152025-03-07T22:34:03.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9162025-03-07T22:34:03.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9172025-03-07T22:34:03.967ZWARNnexus (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set file = nexus/src/app/background/tasks/blueprint_load.rs:69
9182025-03-07T22:34:03.967ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 8.34343ms iteration = 2 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
9192025-03-07T22:34:04.007ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9202025-03-07T22:34:04.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9212025-03-07T22:34:04.046ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 86.523521ms iteration = 2 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
9222025-03-07T22:34:05.536ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9232025-03-07T22:34:05.536ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9242025-03-07T22:34:05.538ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9252025-03-07T22:34:05.538ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9262025-03-07T22:34:05.540ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9272025-03-07T22:34:05.542ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, lookup_type: ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec) } result = Ok(())
9282025-03-07T22:34:05.542ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, time_created: 2025-03-07T22:33:53.849453Z, time_modified: 2025-03-07T22:33:53.849453Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
9292025-03-07T22:34:07.542ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9302025-03-07T22:34:07.543ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9312025-03-07T22:34:07.544ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9322025-03-07T22:34:07.544ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9332025-03-07T22:34:07.546ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9342025-03-07T22:34:07.549ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, lookup_type: ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec) } result = Ok(())
9352025-03-07T22:34:07.549ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, time_created: 2025-03-07T22:33:53.849453Z, time_modified: 2025-03-07T22:33:53.849453Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
9362025-03-07T22:34:09.550ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9372025-03-07T22:34:09.550ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9382025-03-07T22:34:09.552ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9392025-03-07T22:34:09.552ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9402025-03-07T22:34:09.553ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9412025-03-07T22:34:09.556ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, lookup_type: ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec) } result = Ok(())
9422025-03-07T22:34:09.556ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, time_created: 2025-03-07T22:33:53.849453Z, time_modified: 2025-03-07T22:33:53.849453Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
9432025-03-07T22:34:11.556ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9442025-03-07T22:34:11.556ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9452025-03-07T22:34:11.558ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9462025-03-07T22:34:11.558ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9472025-03-07T22:34:11.560ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9482025-03-07T22:34:11.562ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, lookup_type: ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec) } result = Ok(())
9492025-03-07T22:34:11.562ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, time_created: 2025-03-07T22:33:53.849453Z, time_modified: 2025-03-07T22:33:53.849453Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
9502025-03-07T22:34:13.563ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9512025-03-07T22:34:13.564ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9522025-03-07T22:34:13.566ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9532025-03-07T22:34:13.566ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9542025-03-07T22:34:13.568ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9552025-03-07T22:34:13.570ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, lookup_type: ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec) } result = Ok(())
9562025-03-07T22:34:13.570ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, time_created: 2025-03-07T22:33:53.849453Z, time_modified: 2025-03-07T22:33:53.849453Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
9572025-03-07T22:34:13.959ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 3 reason = Timeout
9582025-03-07T22:34:13.959ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 3 reason = Timeout
9592025-03-07T22:34:13.959ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9602025-03-07T22:34:13.959ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9612025-03-07T22:34:13.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9622025-03-07T22:34:13.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9632025-03-07T22:34:13.961ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9642025-03-07T22:34:13.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9652025-03-07T22:34:13.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9662025-03-07T22:34:13.964ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9672025-03-07T22:34:13.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9682025-03-07T22:34:13.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9692025-03-07T22:34:13.965ZWARNnexus (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set file = nexus/src/app/background/tasks/blueprint_load.rs:69
9702025-03-07T22:34:13.965ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.834436ms iteration = 3 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
9712025-03-07T22:34:13.993ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.654523ms iteration = 3 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
9722025-03-07T22:34:15.570ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9732025-03-07T22:34:15.571ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9742025-03-07T22:34:15.573ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9752025-03-07T22:34:15.573ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9762025-03-07T22:34:15.574ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9772025-03-07T22:34:15.576ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, lookup_type: ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec) } result = Ok(())
9782025-03-07T22:34:15.576ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, time_created: 2025-03-07T22:33:53.849453Z, time_modified: 2025-03-07T22:33:53.849453Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
9792025-03-07T22:34:17.577ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9802025-03-07T22:34:17.578ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9812025-03-07T22:34:17.579ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9822025-03-07T22:34:17.579ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9832025-03-07T22:34:17.581ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9842025-03-07T22:34:17.583ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, lookup_type: ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec) } result = Ok(())
9852025-03-07T22:34:17.583ZINFOnexus: Still waiting for rack initialization: Rack { identity: RackIdentity { id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, time_created: 2025-03-07T22:33:53.849453Z, time_modified: 2025-03-07T22:33:53.849453Z }, initialized: false, tuf_base_url: None, rack_subnet: Some(V6(Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 })) } file = nexus/src/app/rack.rs:770
9862025-03-07T22:34:19.585ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9872025-03-07T22:34:19.585ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9882025-03-07T22:34:19.586ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {} }
9892025-03-07T22:34:19.587ZDEBGnexus (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
9902025-03-07T22:34:19.588ZDEBGnexus (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9912025-03-07T22:34:19.590ZDEBGnexus (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, lookup_type: ById(82e3cdb7-0f87-46cf-98cd-832919b5c7ec) } result = Ok(())
9922025-03-07T22:34:19.590ZINFOnexus: Rack initialized file = nexus/src/app/rack.rs:767
9932025-03-07T22:34:19.590ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9942025-03-07T22:34:19.591ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9952025-03-07T22:34:19.592ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9962025-03-07T22:34:19.592ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9972025-03-07T22:34:19.594ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
9982025-03-07T22:34:19.594ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
9992025-03-07T22:34:19.598ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10002025-03-07T22:34:19.598ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10012025-03-07T22:34:19.599ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10022025-03-07T22:34:19.599ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10032025-03-07T22:34:19.600ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10042025-03-07T22:34:19.601ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10052025-03-07T22:34:19.601ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10062025-03-07T22:34:19.604ZDEBGnexus (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
10072025-03-07T22:34:19.604ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10082025-03-07T22:34:19.605ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10092025-03-07T22:34:19.606ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10102025-03-07T22:34:19.606ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10112025-03-07T22:34:19.607ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10122025-03-07T22:34:19.607ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10132025-03-07T22:34:19.608ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10142025-03-07T22:34:19.611ZDEBGnexus (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
10152025-03-07T22:34:19.611ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10162025-03-07T22:34:19.611ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10172025-03-07T22:34:19.616ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10182025-03-07T22:34:19.616ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10192025-03-07T22:34:19.617ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10202025-03-07T22:34:19.617ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10212025-03-07T22:34:19.618ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10222025-03-07T22:34:19.619ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10232025-03-07T22:34:19.620ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10242025-03-07T22:34:19.620ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10252025-03-07T22:34:19.621ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10262025-03-07T22:34:19.621ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10272025-03-07T22:34:19.622ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10282025-03-07T22:34:19.622ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10292025-03-07T22:34:19.623ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10302025-03-07T22:34:19.626ZDEBGnexus (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
10312025-03-07T22:34:19.626ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10322025-03-07T22:34:19.626ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10332025-03-07T22:34:19.627ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10342025-03-07T22:34:19.628ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10352025-03-07T22:34:19.629ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10362025-03-07T22:34:19.629ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10372025-03-07T22:34:19.631ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10382025-03-07T22:34:19.631ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10392025-03-07T22:34:19.632ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10402025-03-07T22:34:19.633ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10412025-03-07T22:34:19.633ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10422025-03-07T22:34:19.633ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10432025-03-07T22:34:19.635ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10442025-03-07T22:34:19.635ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10452025-03-07T22:34:19.636ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10462025-03-07T22:34:19.639ZDEBGnexus (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
10472025-03-07T22:34:19.639ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10482025-03-07T22:34:19.639ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10492025-03-07T22:34:19.640ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10502025-03-07T22:34:19.641ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10512025-03-07T22:34:19.642ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10522025-03-07T22:34:19.642ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10532025-03-07T22:34:19.643ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10542025-03-07T22:34:19.646ZDEBGnexus (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
10552025-03-07T22:34:19.646ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10562025-03-07T22:34:19.646ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10572025-03-07T22:34:19.650ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10582025-03-07T22:34:19.650ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10592025-03-07T22:34:19.652ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10602025-03-07T22:34:19.652ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10612025-03-07T22:34:19.653ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10622025-03-07T22:34:19.653ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10632025-03-07T22:34:19.655ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10642025-03-07T22:34:19.655ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10652025-03-07T22:34:19.656ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10662025-03-07T22:34:19.656ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10672025-03-07T22:34:19.657ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10682025-03-07T22:34:19.658ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10692025-03-07T22:34:19.659ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10702025-03-07T22:34:19.659ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10712025-03-07T22:34:19.660ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10722025-03-07T22:34:19.663ZDEBGnexus (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
10732025-03-07T22:34:19.663ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10742025-03-07T22:34:19.663ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10752025-03-07T22:34:19.665ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10762025-03-07T22:34:19.665ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10772025-03-07T22:34:19.666ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10782025-03-07T22:34:19.666ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10792025-03-07T22:34:19.667ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10802025-03-07T22:34:19.670ZDEBGnexus (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
10812025-03-07T22:34:19.670ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10822025-03-07T22:34:19.671ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10832025-03-07T22:34:19.682ZDEBGnexus (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 867d8023-aff2-4521-891f-07d5f45be915, kind: Service { id: dfe169a3-7536-4b54-9593-8815d6e81dd4 }, name: Name("nexus-dfe169a3-7536-4b54-9593-8815d6e81dd4"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 239])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8980b304-809c-4233-b9df-dd6e12e7c2f3, kind: Service { id: c0f69897-7f92-43db-ad47-96c3c4721128 }, name: Name("nexus-c0f69897-7f92-43db-ad47-96c3c4721128"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 221, 186])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: cb82146b-41bd-465f-8803-0f44afcd8f6e, kind: Service { id: 0f5c512c-2fda-4cff-bbad-41e8a283d9ff }, name: Name("nexus-0f5c512c-2fda-4cff-bbad-41e8a283d9ff"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 140, 207])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 00c422b8-3a81-4d2c-bc3f-fc426f6f9897, kind: Service { id: fbf58a59-609b-4fb3-b792-2c09f3f7b776 }, name: Name("external-dns-fbf58a59-609b-4fb3-b792-2c09f3f7b776"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 149, 171])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 066589e5-6cdf-4b01-9f39-98f2c968e6c2, kind: Service { id: 732b0e55-5ccf-4068-b771-1299f872f618 }, name: Name("external-dns-732b0e55-5ccf-4068-b771-1299f872f618"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
10842025-03-07T22:34:19.682ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10852025-03-07T22:34:19.682ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10862025-03-07T22:34:19.683ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10872025-03-07T22:34:19.685ZDEBGnexus (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
10882025-03-07T22:34:19.685ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10892025-03-07T22:34:19.685ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10902025-03-07T22:34:19.695ZDEBGnexus (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
10912025-03-07T22:34:19.702ZDEBGnexus (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 00c422b8-3a81-4d2c-bc3f-fc426f6f9897, kind: Service { id: fbf58a59-609b-4fb3-b792-2c09f3f7b776 }, name: Name("external-dns-fbf58a59-609b-4fb3-b792-2c09f3f7b776"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 149, 171])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 066589e5-6cdf-4b01-9f39-98f2c968e6c2, kind: Service { id: 732b0e55-5ccf-4068-b771-1299f872f618 }, name: Name("external-dns-732b0e55-5ccf-4068-b771-1299f872f618"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 867d8023-aff2-4521-891f-07d5f45be915, kind: Service { id: dfe169a3-7536-4b54-9593-8815d6e81dd4 }, name: Name("nexus-dfe169a3-7536-4b54-9593-8815d6e81dd4"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 239])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8980b304-809c-4233-b9df-dd6e12e7c2f3, kind: Service { id: c0f69897-7f92-43db-ad47-96c3c4721128 }, name: Name("nexus-c0f69897-7f92-43db-ad47-96c3c4721128"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 221, 186])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: cb82146b-41bd-465f-8803-0f44afcd8f6e, kind: Service { id: 0f5c512c-2fda-4cff-bbad-41e8a283d9ff }, name: Name("nexus-0f5c512c-2fda-4cff-bbad-41e8a283d9ff"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 140, 207])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
10922025-03-07T22:34:19.702ZDEBGnexus (InternalApi): resolved 2 rules for sleds
10932025-03-07T22:34:19.757ZDEBGnexus (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, time_created: 2025-03-07T22:34:17.183069Z, time_modified: 2025-03-07T22:34:17.183069Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, is_scrimlet: true, serial_number: "buskin", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
10942025-03-07T22:34:19.757ZDEBGnexus (InternalApi): sending firewall rules to sled agents
10952025-03-07T22:34:19.757ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10962025-03-07T22:34:19.757ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10972025-03-07T22:34:19.759ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {} }
10982025-03-07T22:34:19.759ZDEBGnexus (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10992025-03-07T22:34:19.761ZDEBGnexus (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11002025-03-07T22:34:19.763ZDEBGnexus (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
11012025-03-07T22:34:19.795ZDEBGnexus (InternalApi): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
11022025-03-07T22:34:19.796ZDEBGnexus (InternalApi): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "59643b1b-a651-46c4-82f9-7abeabad3a58", "date": "Fri, 07 Mar 2025 22:34:19 GMT"} })
11032025-03-07T22:34:19.796ZINFOnexus (InternalApi): updated firewall rules on 1 sleds file = nexus/networking/src/firewall_rules.rs:496
11042025-03-07T22:34:19.796ZINFOnexus: plumbed initial IP allowlist file = nexus/src/app/allow_list.rs:161
11052025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 4 reason = Timeout
11062025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 4 reason = Timeout
11072025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11082025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11092025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
11102025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
11112025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
11122025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
11132025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
11142025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11152025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
11162025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
11172025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 2 reason = Timeout
11182025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11192025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
11202025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 2 reason = Timeout
11212025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11222025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 2 reason = Timeout
11232025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
11242025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11252025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
11262025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11272025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Timeout
11282025-03-07T22:34:23.960ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
11292025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
11302025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11312025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
11322025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
11332025-03-07T22:34:23.960ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11342025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
11352025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11362025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
11372025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11382025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11392025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11402025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
11412025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
11422025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
11432025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11442025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11452025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11462025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11472025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 2 reason = Timeout
11482025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 2 reason = Timeout
11492025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
11502025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
11512025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 2 reason = Timeout
11522025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
11532025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 2 reason = Timeout
11542025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
11552025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 2 reason = Timeout
11562025-03-07T22:34:23.961ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
11572025-03-07T22:34:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
11582025-03-07T22:34:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11592025-03-07T22:34:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
11602025-03-07T22:34:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
11612025-03-07T22:34:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
11622025-03-07T22:34:23.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11632025-03-07T22:34:23.964ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.921468ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
11642025-03-07T22:34:23.964ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11652025-03-07T22:34:23.965ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11662025-03-07T22:34:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
11672025-03-07T22:34:23.966ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11682025-03-07T22:34:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
11692025-03-07T22:34:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
11702025-03-07T22:34:23.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11712025-03-07T22:34:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11722025-03-07T22:34:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11732025-03-07T22:34:23.967ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11742025-03-07T22:34:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11752025-03-07T22:34:23.969ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 822, tv_nsec: 326761739 } })
11762025-03-07T22:34:23.969ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
11772025-03-07T22:34:23.970ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11782025-03-07T22:34:23.970ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11792025-03-07T22:34:23.971ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11802025-03-07T22:34:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
11812025-03-07T22:34:23.973ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11822025-03-07T22:34:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
11832025-03-07T22:34:23.973ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
11842025-03-07T22:34:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11852025-03-07T22:34:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
11862025-03-07T22:34:23.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
11872025-03-07T22:34:23.974ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11882025-03-07T22:34:23.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11892025-03-07T22:34:23.978ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11902025-03-07T22:34:23.978ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11912025-03-07T22:34:23.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11922025-03-07T22:34:23.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
11932025-03-07T22:34:23.980ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 822, tv_nsec: 337819215 } })
11942025-03-07T22:34:23.980ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11952025-03-07T22:34:23.980ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
11962025-03-07T22:34:23.980ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 822, tv_nsec: 338425311 } })
11972025-03-07T22:34:23.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11982025-03-07T22:34:23.981ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
11992025-03-07T22:34:23.992ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12002025-03-07T22:34:23.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12012025-03-07T22:34:24.015ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12022025-03-07T22:34:24.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12032025-03-07T22:34:24.016ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12042025-03-07T22:34:24.016ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
12052025-03-07T22:34:24.016ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
12062025-03-07T22:34:24.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
12072025-03-07T22:34:24.018ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12082025-03-07T22:34:24.021ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12092025-03-07T22:34:24.021ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12102025-03-07T22:34:24.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12112025-03-07T22:34:24.023ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12122025-03-07T22:34:24.024ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12132025-03-07T22:34:24.025ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12142025-03-07T22:34:24.025ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12152025-03-07T22:34:24.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12162025-03-07T22:34:24.027ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
12172025-03-07T22:34:24.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
12182025-03-07T22:34:24.028ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 66.711751ms iteration = 2 status = {"errors":[],"finish_invoked_ok":[]}
12192025-03-07T22:34:24.029ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
12202025-03-07T22:34:24.029ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12212025-03-07T22:34:24.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12222025-03-07T22:34:24.030ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 69.061974ms iteration = 2 status = {"errors":[],"requests_created_ok":[]}
12232025-03-07T22:34:24.032ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12242025-03-07T22:34:24.032ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 70.502137ms iteration = 2 status = {"errors":[],"garbage_collect_requested":[]}
12252025-03-07T22:34:24.032ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 71.850581ms iteration = 2 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
12262025-03-07T22:34:24.034ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12272025-03-07T22:34:24.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
12282025-03-07T22:34:24.035ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
12292025-03-07T22:34:24.035ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12302025-03-07T22:34:24.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12312025-03-07T22:34:24.038ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
12322025-03-07T22:34:24.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
12332025-03-07T22:34:24.038ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
12342025-03-07T22:34:24.038ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 78.162991ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
12352025-03-07T22:34:24.041ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
12362025-03-07T22:34:24.041ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
12372025-03-07T22:34:24.041ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bed97e9b-b441-47c7-8d34-3ce82b88259d", "content-length": "26", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
12382025-03-07T22:34:24.041ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
12392025-03-07T22:34:24.044ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
12402025-03-07T22:34:24.046ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
12412025-03-07T22:34:24.046ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
12422025-03-07T22:34:24.046ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5dd3b1a-b91d-4d20-9b64-c3b5d6a02076", "content-length": "26", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
12432025-03-07T22:34:24.046ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
12442025-03-07T22:34:24.048ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
12452025-03-07T22:34:24.054ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12462025-03-07T22:34:24.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12472025-03-07T22:34:24.061ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
12482025-03-07T22:34:24.061ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
12492025-03-07T22:34:24.062ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3547cfd1-2436-40ab-be4d-d4abe6209590", "content-length": "26", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
12502025-03-07T22:34:24.062ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
12512025-03-07T22:34:24.064ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
12522025-03-07T22:34:24.073ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 112.40042ms iteration = 4 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12532025-03-07T22:34:24.081ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12542025-03-07T22:34:24.081ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
12552025-03-07T22:34:24.081ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
12562025-03-07T22:34:24.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
12572025-03-07T22:34:24.091ZINFOnexus (BackgroundTasks): found new target blueprint (first find) background_task = blueprint_loader file = nexus/src/app/background/tasks/blueprint_load.rs:88 target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
12582025-03-07T22:34:24.091ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 2 reason = Dependency
12592025-03-07T22:34:24.091ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
12602025-03-07T22:34:24.091ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 19.91\u{b5}s iteration = 2 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
12612025-03-07T22:34:24.091ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 131.081919ms iteration = 4 status = {"enabled":false,"status":"first target blueprint","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z","time_found":"2025-03-07T22:34:24.091656966Z"}
12622025-03-07T22:34:24.091ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 2 reason = Dependency
12632025-03-07T22:34:24.091ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12642025-03-07T22:34:24.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12652025-03-07T22:34:24.092ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12662025-03-07T22:34:24.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12672025-03-07T22:34:24.092ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12682025-03-07T22:34:24.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12692025-03-07T22:34:24.092ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12702025-03-07T22:34:24.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12712025-03-07T22:34:24.092ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12722025-03-07T22:34:24.093ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12732025-03-07T22:34:24.094ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12742025-03-07T22:34:24.096ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12752025-03-07T22:34:24.096ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12762025-03-07T22:34:24.097ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12772025-03-07T22:34:24.097ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12782025-03-07T22:34:24.099ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12792025-03-07T22:34:24.099ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12802025-03-07T22:34:24.099ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12812025-03-07T22:34:24.099ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12822025-03-07T22:34:24.101ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 139.227527ms iteration = 2 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
12832025-03-07T22:34:24.101ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12842025-03-07T22:34:24.102ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12852025-03-07T22:34:24.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12862025-03-07T22:34:24.102ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12872025-03-07T22:34:24.103ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12882025-03-07T22:34:24.104ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
12892025-03-07T22:34:24.104ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12902025-03-07T22:34:24.104ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12912025-03-07T22:34:24.105ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12922025-03-07T22:34:24.105ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12932025-03-07T22:34:24.108ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12942025-03-07T22:34:24.108ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12952025-03-07T22:34:24.109ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12962025-03-07T22:34:24.114ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12972025-03-07T22:34:24.117ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
12982025-03-07T22:34:24.117ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12992025-03-07T22:34:24.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13002025-03-07T22:34:24.147ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 187.236199ms iteration = 2 status = {}
13012025-03-07T22:34:24.154ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13022025-03-07T22:34:24.155ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13032025-03-07T22:34:24.155ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
13042025-03-07T22:34:24.155ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
13052025-03-07T22:34:24.155ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 194.763772ms iteration = 2 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
13062025-03-07T22:34:24.159ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13072025-03-07T22:34:24.160ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13082025-03-07T22:34:24.170ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
13092025-03-07T22:34:24.171ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f46fdfd-1fd0-4b28-aa62-da054b62326f", "content-length": "1", "date": "Fri, 07 Mar 2025 22:34:23 GMT"} })
13102025-03-07T22:34:24.172ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
13112025-03-07T22:34:24.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
13122025-03-07T22:34:24.179ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 217.637651ms iteration = 2 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
13132025-03-07T22:34:24.195ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 234.874926ms iteration = 3 status = {"physical_disks_added":0}
13142025-03-07T22:34:24.197ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::5]:32222/node/id
13152025-03-07T22:34:24.215ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
13162025-03-07T22:34:24.216ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
13172025-03-07T22:34:24.217ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "73ac9aa6-5ec7-4442-9061-5486bd4aa7b6", "content-length": "312", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
13182025-03-07T22:34:24.219ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13192025-03-07T22:34:24.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13202025-03-07T22:34:24.248ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::6]:32222/node/id
13212025-03-07T22:34:24.251ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13222025-03-07T22:34:24.251ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13232025-03-07T22:34:24.252ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13242025-03-07T22:34:24.253ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
13252025-03-07T22:34:24.253ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13262025-03-07T22:34:24.253ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 292.43163ms iteration = 2 status = null
13272025-03-07T22:34:24.253ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13282025-03-07T22:34:24.255ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13292025-03-07T22:34:24.255ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13302025-03-07T22:34:24.258ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
13312025-03-07T22:34:24.258ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13322025-03-07T22:34:24.258ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
13332025-03-07T22:34:24.258ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13342025-03-07T22:34:24.258ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13352025-03-07T22:34:24.258ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13362025-03-07T22:34:24.265ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13372025-03-07T22:34:24.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13382025-03-07T22:34:24.282ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
13392025-03-07T22:34:24.282ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 321.458969ms iteration = 2 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
13402025-03-07T22:34:24.290ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::7]:32222/node/id
13412025-03-07T22:34:24.302ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
13422025-03-07T22:34:24.303ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "9834db65-f9d8-482d-a299-a7d2378712b2", "content-length": "691", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
13432025-03-07T22:34:24.305ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
13442025-03-07T22:34:24.305ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
13452025-03-07T22:34:24.305ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
13462025-03-07T22:34:24.306ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 345.46545ms iteration = 2 status = {}
13472025-03-07T22:34:24.328ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::3]:32222/node/id
13482025-03-07T22:34:24.330ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13492025-03-07T22:34:24.331ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13502025-03-07T22:34:24.331ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
13512025-03-07T22:34:24.331ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13522025-03-07T22:34:24.331ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13532025-03-07T22:34:24.351ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13542025-03-07T22:34:24.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13552025-03-07T22:34:24.362ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::4]:32222/node/id
13562025-03-07T22:34:24.371ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13572025-03-07T22:34:24.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13582025-03-07T22:34:24.379ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
13592025-03-07T22:34:24.383ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf525323-4f37-491c-bdcf-b7fd34867cff", "content-length": "134", "date": "Fri, 07 Mar 2025 22:34:23 GMT"} })
13602025-03-07T22:34:24.383ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
13612025-03-07T22:34:24.383ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
13622025-03-07T22:34:24.384ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
13632025-03-07T22:34:24.384ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
13642025-03-07T22:34:24.384ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13652025-03-07T22:34:24.384ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13662025-03-07T22:34:24.384ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13672025-03-07T22:34:24.384ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
13682025-03-07T22:34:24.386ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13692025-03-07T22:34:24.386ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
13702025-03-07T22:34:24.389ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13712025-03-07T22:34:24.389ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
13722025-03-07T22:34:24.389ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13732025-03-07T22:34:24.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13742025-03-07T22:34:24.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13752025-03-07T22:34:24.401ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
13762025-03-07T22:34:24.404ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "457ce584-985c-4d68-a1bd-d5ce162d817d", "content-length": "2", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
13772025-03-07T22:34:24.404ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
13782025-03-07T22:34:24.404ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "c99dc99e-9aa1-438e-a047-01cc373a400d", "content-length": "52", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
13792025-03-07T22:34:24.404ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
13802025-03-07T22:34:24.404ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
13812025-03-07T22:34:24.413ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13822025-03-07T22:34:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13832025-03-07T22:34:24.426ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13842025-03-07T22:34:24.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13852025-03-07T22:34:24.436ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
13862025-03-07T22:34:24.436ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
13872025-03-07T22:34:24.440ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 822, tv_nsec: 798145968 } })
13882025-03-07T22:34:24.440ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = service_zone_nat_tracker file = nexus/src/app/mod.rs:1171
13892025-03-07T22:34:24.494ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = service_zone_nat_tracker file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
13902025-03-07T22:34:24.494ZDEBGnexus (MgsClient): client request background_task = service_zone_nat_tracker body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13912025-03-07T22:34:24.494ZDEBGnexus (MgsClient): client response background_task = service_zone_nat_tracker result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ec60d60-47c9-44ae-919b-1590e240a648", "content-length": "26", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
13922025-03-07T22:34:24.494ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = service_zone_nat_tracker file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
13932025-03-07T22:34:24.495ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = service_zone_nat_tracker file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
13942025-03-07T22:34:24.534ZDEBGnexus (DpdClient): client request background_task = service_zone_nat_tracker body = None method = POST uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/trigger
13952025-03-07T22:34:24.535ZDEBGnexus (DpdClient): client response background_task = service_zone_nat_tracker result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/trigger", status: 200, headers: {"content-type": "application/json", "x-request-id": "05e531b5-f859-4031-99f5-897ba4a10c83", "content-length": "4", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
13962025-03-07T22:34:24.536ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 576.123297ms iteration = 2 status = 6
13972025-03-07T22:34:24.596ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "db8edd83-077a-4e39-983d-d217d255952c", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
13982025-03-07T22:34:24.596ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
13992025-03-07T22:34:24.596ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
14002025-03-07T22:34:24.596ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "067e9db9-bba8-49dc-95a9-2d4bfdefdc81", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
14012025-03-07T22:34:24.596ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14022025-03-07T22:34:24.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14032025-03-07T22:34:24.598ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "8712139a-f296-4cc4-9b39-4bc89a509ea6", "content-length": "539", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
14042025-03-07T22:34:24.599ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14052025-03-07T22:34:24.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14062025-03-07T22:34:24.600ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14072025-03-07T22:34:24.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14082025-03-07T22:34:24.605ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14092025-03-07T22:34:24.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14102025-03-07T22:34:24.609ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14112025-03-07T22:34:24.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14122025-03-07T22:34:24.612ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14132025-03-07T22:34:24.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14142025-03-07T22:34:24.613ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14152025-03-07T22:34:24.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14162025-03-07T22:34:24.615ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14172025-03-07T22:34:24.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14182025-03-07T22:34:24.617ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14192025-03-07T22:34:24.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14202025-03-07T22:34:24.618ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14212025-03-07T22:34:24.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14222025-03-07T22:34:24.619ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14232025-03-07T22:34:24.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14242025-03-07T22:34:24.620ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14252025-03-07T22:34:24.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14262025-03-07T22:34:24.621ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14272025-03-07T22:34:24.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14282025-03-07T22:34:24.624ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14292025-03-07T22:34:24.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14302025-03-07T22:34:24.629ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14312025-03-07T22:34:24.633ZWARNnexus (BackgroundTasks): no bootstore config found in db background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1229 key = network_key rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
14322025-03-07T22:34:24.633ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14332025-03-07T22:34:24.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14342025-03-07T22:34:24.634ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
14352025-03-07T22:34:24.634ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14362025-03-07T22:34:24.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14372025-03-07T22:34:24.636ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14382025-03-07T22:34:24.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14392025-03-07T22:34:24.638ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14402025-03-07T22:34:24.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14412025-03-07T22:34:24.640ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14422025-03-07T22:34:24.644ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
14432025-03-07T22:34:24.644ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14442025-03-07T22:34:24.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14452025-03-07T22:34:24.648ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14462025-03-07T22:34:24.648ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14472025-03-07T22:34:24.651ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14482025-03-07T22:34:24.651ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14492025-03-07T22:34:24.653ZINFOnexus (BackgroundTasks): updating bootstore config background_task = switch_port_config_manager config = EarlyNetworkConfig { body: EarlyNetworkConfigBody { ntp_servers: ["0.pool.ntp.org"], rack_network_config: Some(RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], autoneg: false, bgp_peers: [], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, rib_priority: None, vlan_id: None }], switch: Switch0, tx_eq: None, uplink_port_fec: Some(None), uplink_port_speed: Speed40G }], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 } }) }, generation: 2, schema_version: 2 } file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1286 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
14502025-03-07T22:34:24.653ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14512025-03-07T22:34:24.653ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14522025-03-07T22:34:24.655ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14532025-03-07T22:34:24.655ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = PUT rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
14542025-03-07T22:34:24.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14552025-03-07T22:34:24.658ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14562025-03-07T22:34:24.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14572025-03-07T22:34:24.659ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 204, headers: {"x-request-id": "8ee8ee89-4dce-4677-8359-916e45283752", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
14582025-03-07T22:34:24.659ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14592025-03-07T22:34:24.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14602025-03-07T22:34:24.659ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14612025-03-07T22:34:24.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14622025-03-07T22:34:24.661ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14632025-03-07T22:34:24.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14642025-03-07T22:34:24.663ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14652025-03-07T22:34:24.668ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ById(001de000-c470-4000-8000-000000000002) } result = Ok(())
14662025-03-07T22:34:24.668ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14672025-03-07T22:34:24.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14682025-03-07T22:34:24.670ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14692025-03-07T22:34:24.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14702025-03-07T22:34:24.673ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14712025-03-07T22:34:24.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14722025-03-07T22:34:24.676ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14732025-03-07T22:34:24.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14742025-03-07T22:34:24.677ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 716.858399ms iteration = 2 status = {}
14752025-03-07T22:34:24.678ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14762025-03-07T22:34:24.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14772025-03-07T22:34:24.680ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14782025-03-07T22:34:24.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14792025-03-07T22:34:24.682ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14802025-03-07T22:34:24.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14812025-03-07T22:34:24.684ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14822025-03-07T22:34:24.688ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ById(001de000-c470-4000-8000-000000000001) } result = Ok(())
14832025-03-07T22:34:24.688ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14842025-03-07T22:34:24.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14852025-03-07T22:34:24.691ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14862025-03-07T22:34:24.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14872025-03-07T22:34:24.693ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14882025-03-07T22:34:24.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14892025-03-07T22:34:24.694ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14902025-03-07T22:34:24.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14912025-03-07T22:34:24.696ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14922025-03-07T22:34:24.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14932025-03-07T22:34:24.698ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14942025-03-07T22:34:24.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14952025-03-07T22:34:24.699ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14962025-03-07T22:34:24.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14972025-03-07T22:34:24.701ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14982025-03-07T22:34:24.704ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-c470-4000-8000-000000000003, lookup_type: ById(001de000-c470-4000-8000-000000000003) } result = Ok(())
14992025-03-07T22:34:24.704ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15002025-03-07T22:34:24.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15012025-03-07T22:34:24.706ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15022025-03-07T22:34:24.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15032025-03-07T22:34:24.707ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15042025-03-07T22:34:24.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15052025-03-07T22:34:24.709ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15062025-03-07T22:34:24.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15072025-03-07T22:34:24.711ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15082025-03-07T22:34:24.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15092025-03-07T22:34:24.713ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15102025-03-07T22:34:24.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15112025-03-07T22:34:24.714ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15122025-03-07T22:34:24.714ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15132025-03-07T22:34:24.716ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15142025-03-07T22:34:24.719ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-c470-4000-8000-000000000003, lookup_type: ByName("boundary-ntp") } result = Ok(())
15152025-03-07T22:34:24.720ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15162025-03-07T22:34:24.720ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15172025-03-07T22:34:24.721ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15182025-03-07T22:34:24.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15192025-03-07T22:34:24.723ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15202025-03-07T22:34:24.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15212025-03-07T22:34:24.724ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15222025-03-07T22:34:24.724ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15232025-03-07T22:34:24.726ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15242025-03-07T22:34:24.726ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15252025-03-07T22:34:24.727ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15262025-03-07T22:34:24.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15272025-03-07T22:34:24.729ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15282025-03-07T22:34:24.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15292025-03-07T22:34:24.730ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15302025-03-07T22:34:24.734ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
15312025-03-07T22:34:24.735ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15322025-03-07T22:34:24.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15332025-03-07T22:34:24.736ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15342025-03-07T22:34:24.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15352025-03-07T22:34:24.738ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15362025-03-07T22:34:24.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15372025-03-07T22:34:24.739ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15382025-03-07T22:34:24.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15392025-03-07T22:34:24.742ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15402025-03-07T22:34:24.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15412025-03-07T22:34:24.743ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15422025-03-07T22:34:24.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15432025-03-07T22:34:24.745ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15442025-03-07T22:34:24.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15452025-03-07T22:34:24.747ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15462025-03-07T22:34:24.750ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
15472025-03-07T22:34:24.750ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15482025-03-07T22:34:24.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15492025-03-07T22:34:24.752ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15502025-03-07T22:34:24.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15512025-03-07T22:34:24.754ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15522025-03-07T22:34:24.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15532025-03-07T22:34:24.759ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15542025-03-07T22:34:24.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15552025-03-07T22:34:24.762ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15562025-03-07T22:34:24.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15572025-03-07T22:34:24.763ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15582025-03-07T22:34:24.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15592025-03-07T22:34:24.765ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15602025-03-07T22:34:24.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15612025-03-07T22:34:24.766ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15622025-03-07T22:34:24.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = InternetGateway { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000002, lookup_type: ByName("default") } result = Ok(())
15632025-03-07T22:34:24.773ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
15642025-03-07T22:34:24.775ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "2db09892-2db7-4908-8b54-c738da8db214", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
15652025-03-07T22:34:24.777ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 816.272164ms iteration = 2 status = {}
15662025-03-07T22:34:25.439ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::5]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bd25a3e-403f-4b07-97fa-9670e796f366", "content-length": "64", "date": "Fri, 07 Mar 2025 22:34:25 GMT"} })
15672025-03-07T22:34:25.440ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
15682025-03-07T22:34:25.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
15692025-03-07T22:34:25.442ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15702025-03-07T22:34:25.443ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
15712025-03-07T22:34:25.443ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
15722025-03-07T22:34:25.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
15732025-03-07T22:34:25.462ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::7]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "609909f5-0eba-4c49-874e-be923969006f", "content-length": "64", "date": "Fri, 07 Mar 2025 22:34:24 GMT"} })
15742025-03-07T22:34:25.462ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
15752025-03-07T22:34:25.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
15762025-03-07T22:34:25.464ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15772025-03-07T22:34:25.465ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
15782025-03-07T22:34:25.465ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
15792025-03-07T22:34:25.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
15802025-03-07T22:34:25.489ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::6]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbbb9226-7def-4b73-a810-e9bbd3d26ed7", "content-length": "64", "date": "Fri, 07 Mar 2025 22:34:25 GMT"} })
15812025-03-07T22:34:25.489ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
15822025-03-07T22:34:25.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
15832025-03-07T22:34:25.490ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15842025-03-07T22:34:25.491ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
15852025-03-07T22:34:25.492ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
15862025-03-07T22:34:25.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
15872025-03-07T22:34:25.560ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::4]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5285c81-bb33-4dee-ba92-bc452c08c7ce", "content-length": "64", "date": "Fri, 07 Mar 2025 22:34:25 GMT"} })
15882025-03-07T22:34:25.560ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
15892025-03-07T22:34:25.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
15902025-03-07T22:34:25.562ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15912025-03-07T22:34:25.563ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
15922025-03-07T22:34:25.563ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
15932025-03-07T22:34:25.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
15942025-03-07T22:34:25.587ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::3]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6364578-6918-4644-8205-181afab33273", "content-length": "64", "date": "Fri, 07 Mar 2025 22:34:25 GMT"} })
15952025-03-07T22:34:25.587ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
15962025-03-07T22:34:25.587ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
15972025-03-07T22:34:25.589ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15982025-03-07T22:34:25.590ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
15992025-03-07T22:34:25.590ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
16002025-03-07T22:34:25.590ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16012025-03-07T22:34:25.595ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 1.503933223s iteration = 2 status = {"nsuccess":5}
16022025-03-07T22:34:33.960ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 5 reason = Timeout
16032025-03-07T22:34:33.960ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 5 reason = Timeout
16042025-03-07T22:34:33.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16052025-03-07T22:34:33.960ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16062025-03-07T22:34:33.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16072025-03-07T22:34:33.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16082025-03-07T22:34:33.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16092025-03-07T22:34:33.964ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16102025-03-07T22:34:33.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16112025-03-07T22:34:33.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
16122025-03-07T22:34:33.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16132025-03-07T22:34:33.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16142025-03-07T22:34:33.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16152025-03-07T22:34:33.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16162025-03-07T22:34:33.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16172025-03-07T22:34:33.977ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
16182025-03-07T22:34:33.977ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16192025-03-07T22:34:33.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16202025-03-07T22:34:33.991ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
16212025-03-07T22:34:33.991ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 31.22932ms iteration = 5 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
16222025-03-07T22:34:34.007ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 47.098693ms iteration = 5 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16232025-03-07T22:34:34.662ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:56191
16242025-03-07T22:34:34.684ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56191 req_id = 5bd476c6-da70-4b7d-b88c-17bdf4aefb92 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
16252025-03-07T22:34:34.685ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56191 req_id = 5bd476c6-da70-4b7d-b88c-17bdf4aefb92 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
16262025-03-07T22:34:34.690ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 24236 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56191 req_id = 5bd476c6-da70-4b7d-b88c-17bdf4aefb92 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
16272025-03-07T22:34:34.693ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56191 req_id = 3e9536cb-035a-44fc-a0ce-b649cebba01c roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJiM2FjZDJlYS1jYzU2LTRkNzAtOWVhZS0xMGJiMjQ3Y2JiNjQifX0%3D
16282025-03-07T22:34:34.693ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56191 req_id = 3e9536cb-035a-44fc-a0ce-b649cebba01c resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJiM2FjZDJlYS1jYzU2LTRkNzAtOWVhZS0xMGJiMjQ3Y2JiNjQifX0%3D
16292025-03-07T22:34:34.696ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 4915 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56191 req_id = 3e9536cb-035a-44fc-a0ce-b649cebba01c response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJiM2FjZDJlYS1jYzU2LTRkNzAtOWVhZS0xMGJiMjQ3Y2JiNjQifX0%3D
16302025-03-07T22:34:43.959ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 6 reason = Timeout
16312025-03-07T22:34:43.960ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 6 reason = Timeout
16322025-03-07T22:34:43.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16332025-03-07T22:34:43.960ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16342025-03-07T22:34:43.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16352025-03-07T22:34:43.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16362025-03-07T22:34:43.961ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16372025-03-07T22:34:43.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16382025-03-07T22:34:43.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16392025-03-07T22:34:43.964ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
16402025-03-07T22:34:43.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16412025-03-07T22:34:43.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16422025-03-07T22:34:43.964ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.437732ms iteration = 6 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16432025-03-07T22:34:43.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16442025-03-07T22:34:43.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16452025-03-07T22:34:43.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16462025-03-07T22:34:43.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
16472025-03-07T22:34:43.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16482025-03-07T22:34:43.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16492025-03-07T22:34:43.981ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
16502025-03-07T22:34:43.981ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.7625ms iteration = 6 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
16512025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 7 reason = Timeout
16522025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 4 reason = Timeout
16532025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Timeout
16542025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16552025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16562025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 3 reason = Timeout
16572025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
16582025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16592025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 3 reason = Timeout
16602025-03-07T22:34:53.959ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
16612025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 12.65\u{b5}s iteration = 3 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
16622025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 3 reason = Timeout
16632025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
16642025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Timeout
16652025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Timeout
16662025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
16672025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 3 reason = Timeout
16682025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 3 reason = Timeout
16692025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Timeout
16702025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
16712025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
16722025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
16732025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Timeout
16742025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16752025-03-07T22:34:53.959ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:69 reason = no config
16762025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 8.2\u{b5}s iteration = 3 status = {"error":"no config"}
16772025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 3 reason = Timeout
16782025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 2 reason = Timeout
16792025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 3 reason = Timeout
16802025-03-07T22:34:53.959ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:24:53.959679672 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
16812025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
16822025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
16832025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 3 reason = Timeout
16842025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16852025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 3 reason = Timeout
16862025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 2 reason = Timeout
16872025-03-07T22:34:53.959ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
16882025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16892025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Timeout
16902025-03-07T22:34:53.959ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:69 reason = no config
16912025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
16922025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 11.87\u{b5}s iteration = 3 status = {"error":"no config"}
16932025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 3 reason = Timeout
16942025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 3 reason = Timeout
16952025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16962025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 2 reason = Timeout
16972025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16982025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
16992025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 3 reason = Timeout
17002025-03-07T22:34:53.959ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
17012025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 7 reason = Timeout
17022025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17032025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17042025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Timeout
17052025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
17062025-03-07T22:34:53.959ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17072025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
17082025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
17092025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17102025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17112025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17122025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
17132025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
17142025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
17152025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17162025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
17172025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17182025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17192025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
17202025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
17212025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17222025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17232025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
17242025-03-07T22:34:53.960ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 852, tv_nsec: 318429719 } })
17252025-03-07T22:34:53.960ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 852, tv_nsec: 318444359 } })
17262025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 3 reason = Timeout
17272025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
17282025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 3 reason = Timeout
17292025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
17302025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 2 reason = Timeout
17312025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17322025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17332025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
17342025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 3 reason = Timeout
17352025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 2 reason = Timeout
17362025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 3 reason = Timeout
17372025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
17382025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
17392025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
17402025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 3 reason = Timeout
17412025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
17422025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
17432025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
17442025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
17452025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
17462025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
17472025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
17482025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
17492025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
17502025-03-07T22:34:53.960ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.349593ms iteration = 2 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
17512025-03-07T22:34:53.961ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
17522025-03-07T22:34:53.961ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.525435ms iteration = 2 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
17532025-03-07T22:34:53.962ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17542025-03-07T22:34:53.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17552025-03-07T22:34:53.964ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17562025-03-07T22:34:53.964ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
17572025-03-07T22:34:53.964ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17582025-03-07T22:34:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17592025-03-07T22:34:53.964ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.283211ms iteration = 3 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
17602025-03-07T22:34:53.965ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17612025-03-07T22:34:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
17622025-03-07T22:34:53.966ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17632025-03-07T22:34:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17642025-03-07T22:34:53.966ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 852, tv_nsec: 324252605 } })
17652025-03-07T22:34:53.966ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
17662025-03-07T22:34:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
17672025-03-07T22:34:53.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17682025-03-07T22:34:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17692025-03-07T22:34:53.967ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:24:53.959679672 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
17702025-03-07T22:34:53.967ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 8.220619ms iteration = 2 status = {"expiration":"2025-03-07T22:24:53.959679672Z","pruned":[]}
17712025-03-07T22:34:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
17722025-03-07T22:34:53.967ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
17732025-03-07T22:34:53.968ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17742025-03-07T22:34:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
17752025-03-07T22:34:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17762025-03-07T22:34:53.968ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17772025-03-07T22:34:53.969ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 852, tv_nsec: 327192713 } })
17782025-03-07T22:34:53.969ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
17792025-03-07T22:34:53.969ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17802025-03-07T22:34:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
17812025-03-07T22:34:53.971ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
17822025-03-07T22:34:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
17832025-03-07T22:34:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
17842025-03-07T22:34:53.972ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17852025-03-07T22:34:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17862025-03-07T22:34:53.973ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17872025-03-07T22:34:53.975ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17882025-03-07T22:34:53.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
17892025-03-07T22:34:53.975ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
17902025-03-07T22:34:53.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
17912025-03-07T22:34:53.976ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17922025-03-07T22:34:53.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17932025-03-07T22:34:53.977ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
17942025-03-07T22:34:53.977ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17952025-03-07T22:34:53.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17962025-03-07T22:34:53.980ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 19.956551ms iteration = 2 status = {"errors":[],"found_port_ok":[]}
17972025-03-07T22:34:53.981ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 21.239994ms iteration = 3 status = {"errors":[],"finish_invoked_ok":[]}
17982025-03-07T22:34:53.982ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17992025-03-07T22:34:53.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18002025-03-07T22:34:53.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
18012025-03-07T22:34:53.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18022025-03-07T22:34:53.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18032025-03-07T22:34:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
18042025-03-07T22:34:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18052025-03-07T22:34:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18062025-03-07T22:34:53.983ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18072025-03-07T22:34:53.985ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18082025-03-07T22:34:53.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
18092025-03-07T22:34:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
18102025-03-07T22:34:53.989ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18112025-03-07T22:34:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18122025-03-07T22:34:53.990ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18132025-03-07T22:34:53.990ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18142025-03-07T22:34:53.991ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18152025-03-07T22:34:53.993ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
18162025-03-07T22:34:53.993ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18172025-03-07T22:34:53.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18182025-03-07T22:34:53.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
18192025-03-07T22:34:53.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
18202025-03-07T22:34:53.996ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18212025-03-07T22:34:53.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18222025-03-07T22:34:53.998ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18232025-03-07T22:34:53.999ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18242025-03-07T22:34:54.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
18252025-03-07T22:34:54.002ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18262025-03-07T22:34:54.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18272025-03-07T22:34:54.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
18282025-03-07T22:34:54.002ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18292025-03-07T22:34:54.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18302025-03-07T22:34:54.008ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18312025-03-07T22:34:54.011ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
18322025-03-07T22:34:54.011ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18332025-03-07T22:34:54.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18342025-03-07T22:34:54.014ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
18352025-03-07T22:34:54.014ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
18362025-03-07T22:34:54.014ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
18372025-03-07T22:34:54.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
18382025-03-07T22:34:54.016ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
18392025-03-07T22:34:54.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
18402025-03-07T22:34:54.016ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18412025-03-07T22:34:54.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18422025-03-07T22:34:54.024ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 63.982134ms iteration = 3 status = {"errors":[],"garbage_collect_requested":[]}
18432025-03-07T22:34:54.026ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 66.052604ms iteration = 3 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
18442025-03-07T22:34:54.030ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
18452025-03-07T22:34:54.030ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
18462025-03-07T22:34:54.030ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d069b480-dfac-4fee-93f8-9b98034df65a", "content-length": "26", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
18472025-03-07T22:34:54.030ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18482025-03-07T22:34:54.032ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
18492025-03-07T22:34:54.035ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18502025-03-07T22:34:54.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18512025-03-07T22:34:54.037ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
18522025-03-07T22:34:54.037ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
18532025-03-07T22:34:54.037ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 76.806107ms iteration = 2 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
18542025-03-07T22:34:54.037ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
18552025-03-07T22:34:54.037ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 77.893288ms iteration = 3 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
18562025-03-07T22:34:54.044ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
18572025-03-07T22:34:54.044ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
18582025-03-07T22:34:54.044ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
18592025-03-07T22:34:54.044ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18602025-03-07T22:34:54.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18612025-03-07T22:34:54.051ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
18622025-03-07T22:34:54.051ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
18632025-03-07T22:34:54.051ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 91.392328ms iteration = 2 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
18642025-03-07T22:34:54.054ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
18652025-03-07T22:34:54.054ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
18662025-03-07T22:34:54.054ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
18672025-03-07T22:34:54.054ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18682025-03-07T22:34:54.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18692025-03-07T22:34:54.064ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
18702025-03-07T22:34:54.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
18712025-03-07T22:34:54.087ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
18722025-03-07T22:34:54.087ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
18732025-03-07T22:34:54.087ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f808d41e-0af4-41da-9f2d-9b44cd7cb629", "content-length": "26", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
18742025-03-07T22:34:54.087ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18752025-03-07T22:34:54.089ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
18762025-03-07T22:34:54.095ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
18772025-03-07T22:34:54.095ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
18782025-03-07T22:34:54.096ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "df11390e-7274-4045-9da9-b118f4ee6308", "content-length": "372", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
18792025-03-07T22:34:54.096ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18802025-03-07T22:34:54.096ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18812025-03-07T22:34:54.101ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 852, tv_nsec: 459167181 } })
18822025-03-07T22:34:54.101ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
18832025-03-07T22:34:54.101ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
18842025-03-07T22:34:54.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
18852025-03-07T22:34:54.103ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18862025-03-07T22:34:54.107ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
18872025-03-07T22:34:54.107ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18882025-03-07T22:34:54.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18892025-03-07T22:34:54.126ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 166.282548ms iteration = 3 status = {"errors":[],"requests_created_ok":[]}
18902025-03-07T22:34:54.126ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18912025-03-07T22:34:54.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18922025-03-07T22:34:54.133ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
18932025-03-07T22:34:54.133ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18942025-03-07T22:34:54.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
18952025-03-07T22:34:54.135ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
18962025-03-07T22:34:54.135ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 176.236663ms iteration = 7 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
18972025-03-07T22:34:54.136ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
18982025-03-07T22:34:54.136ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18992025-03-07T22:34:54.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19002025-03-07T22:34:54.162ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19012025-03-07T22:34:54.164ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
19022025-03-07T22:34:54.164ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
19032025-03-07T22:34:54.164ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
19042025-03-07T22:34:54.164ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19052025-03-07T22:34:54.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
19062025-03-07T22:34:54.165ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
19072025-03-07T22:34:54.165ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
19082025-03-07T22:34:54.165ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19092025-03-07T22:34:54.165ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
19102025-03-07T22:34:54.165ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19112025-03-07T22:34:54.166ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19122025-03-07T22:34:54.166ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19132025-03-07T22:34:54.169ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19142025-03-07T22:34:54.169ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19152025-03-07T22:34:54.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19162025-03-07T22:34:54.170ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
19172025-03-07T22:34:54.170ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "a156a31a-41dd-46ec-9676-6c2e6b706169", "content-length": "1", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
19182025-03-07T22:34:54.172ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 212.406371ms iteration = 7 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
19192025-03-07T22:34:54.172ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
19202025-03-07T22:34:54.172ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19212025-03-07T22:34:54.172ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
19222025-03-07T22:34:54.174ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
19232025-03-07T22:34:54.174ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
19242025-03-07T22:34:54.174ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19252025-03-07T22:34:54.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19262025-03-07T22:34:54.176ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 216.50428ms iteration = 3 status = null
19272025-03-07T22:34:54.177ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
19282025-03-07T22:34:54.177ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19292025-03-07T22:34:54.177ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19302025-03-07T22:34:54.177ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19312025-03-07T22:34:54.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19322025-03-07T22:34:54.179ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2af02fd4-ea26-4f7b-8b65-666fe4c4db09", "content-length": "26", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
19332025-03-07T22:34:54.179ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19342025-03-07T22:34:54.179ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 219.340617ms iteration = 3 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
19352025-03-07T22:34:54.180ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
19362025-03-07T22:34:54.181ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19372025-03-07T22:34:54.184ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
19382025-03-07T22:34:54.185ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19392025-03-07T22:34:54.185ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19402025-03-07T22:34:54.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19412025-03-07T22:34:54.188ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
19422025-03-07T22:34:54.189ZINFOnexus (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal file = nexus/src/app/background/tasks/dns_config.rs:87 generation = 1
19432025-03-07T22:34:54.189ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 229.226572ms iteration = 2 status = {"generation":1}
19442025-03-07T22:34:54.189ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 4 reason = Dependency
19452025-03-07T22:34:54.190ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 230.541475ms iteration = 3 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
19462025-03-07T22:34:54.191ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
19472025-03-07T22:34:54.191ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
19482025-03-07T22:34:54.191ZINFOnexus (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external file = nexus/src/app/background/tasks/dns_config.rs:87 generation = 2
19492025-03-07T22:34:54.191ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 231.723586ms iteration = 2 status = {"generation":2}
19502025-03-07T22:34:54.191ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 4 reason = Dependency
19512025-03-07T22:34:54.198ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19522025-03-07T22:34:54.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19532025-03-07T22:34:54.200ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19542025-03-07T22:34:54.203ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
19552025-03-07T22:34:54.203ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19562025-03-07T22:34:54.203ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19572025-03-07T22:34:54.204ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19582025-03-07T22:34:54.206ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
19592025-03-07T22:34:54.206ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19602025-03-07T22:34:54.207ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19612025-03-07T22:34:54.210ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 251.093573ms iteration = 3 status = {}
19622025-03-07T22:34:54.218ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
19632025-03-07T22:34:54.218ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
19642025-03-07T22:34:54.218ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 258.610434ms iteration = 3 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
19652025-03-07T22:34:54.218ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
19662025-03-07T22:34:54.218ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 259.029729ms iteration = 3 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
19672025-03-07T22:34:54.226ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
19682025-03-07T22:34:54.226ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 266.606412ms iteration = 2 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
19692025-03-07T22:34:54.253ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:77
19702025-03-07T22:34:54.253ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 293.4583ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
19712025-03-07T22:34:54.262ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19722025-03-07T22:34:54.262ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19732025-03-07T22:34:54.266ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19742025-03-07T22:34:54.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19752025-03-07T22:34:54.268ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
19762025-03-07T22:34:54.268ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
19772025-03-07T22:34:54.268ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "c8a07d25-8602-4201-9bdc-e908320ba684", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
19782025-03-07T22:34:54.268ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19792025-03-07T22:34:54.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19802025-03-07T22:34:54.269ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19812025-03-07T22:34:54.271ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19822025-03-07T22:34:54.271ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
19832025-03-07T22:34:54.271ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19842025-03-07T22:34:54.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19852025-03-07T22:34:54.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19862025-03-07T22:34:54.274ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 314.848415ms iteration = 3 status = 0
19872025-03-07T22:34:54.276ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19882025-03-07T22:34:54.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19892025-03-07T22:34:54.280ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
19902025-03-07T22:34:54.280ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
19912025-03-07T22:34:54.280ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "3681c2db-521c-43f4-a385-31d5256b31cf", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
19922025-03-07T22:34:54.282ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 322.740751ms iteration = 3 status = {}
19932025-03-07T22:34:54.287ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 327.727959ms iteration = 4 status = {"physical_disks_added":0}
19942025-03-07T22:34:54.288ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
19952025-03-07T22:34:54.293ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
19962025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = / versions = all
19972025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /access versions = all
19982025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /assets/{path} versions = all
19992025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /device/auth versions = all
20002025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /device/confirm versions = all
20012025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /device/success versions = all
20022025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /device/token versions = all
20032025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /device/verify versions = all
20042025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /experimental/v1/probes versions = all
20052025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /experimental/v1/probes versions = all
20062025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /experimental/v1/probes/{probe} versions = all
20072025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /experimental/v1/probes/{probe} versions = all
20082025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /experimental/v1/system/support-bundles versions = all
20092025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /experimental/v1/system/support-bundles versions = all
20102025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
20112025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
20122025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
20132025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
20142025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
20152025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
20162025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
20172025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /images versions = all
20182025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /login versions = all
20192025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /login/{silo_name}/local versions = all
20202025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
20212025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
20222025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
20232025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /lookup/{path} versions = all
20242025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /projects/{path} versions = all
20252025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /projects-new versions = all
20262025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /settings/{path} versions = all
20272025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /system/{path} versions = all
20282025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /utilization versions = all
20292025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/affinity-groups versions = all
20302025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/affinity-groups versions = all
20312025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
20322025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
20332025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
20342025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
20352025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
20362025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
20372025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
20382025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/anti-affinity-groups versions = all
20392025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/anti-affinity-groups versions = all
20402025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
20412025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
20422025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
20432025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
20442025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
20452025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
20462025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
20472025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/certificates versions = all
20482025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/certificates versions = all
20492025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/certificates/{certificate} versions = all
20502025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/certificates/{certificate} versions = all
20512025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/disks versions = all
20522025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/disks versions = all
20532025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/disks/{disk} versions = all
20542025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/disks/{disk} versions = all
20552025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/disks/{disk}/bulk-write versions = all
20562025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
20572025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
20582025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/disks/{disk}/finalize versions = all
20592025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
20602025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/floating-ips versions = all
20612025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/floating-ips versions = all
20622025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
20632025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/floating-ips/{floating_ip} versions = all
20642025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
20652025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
20662025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
20672025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/groups versions = all
20682025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/groups/{group_id} versions = all
20692025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/images versions = all
20702025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/images versions = all
20712025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/images/{image} versions = all
20722025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/images/{image} versions = all
20732025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/images/{image}/demote versions = all
20742025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/images/{image}/promote versions = all
20752025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/instances versions = all
20762025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/instances versions = all
20772025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/instances/{instance} versions = all
20782025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/instances/{instance} versions = all
20792025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/instances/{instance} versions = all
20802025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/instances/{instance}/disks versions = all
20812025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/instances/{instance}/disks/attach versions = all
20822025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/instances/{instance}/disks/detach versions = all
20832025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/instances/{instance}/external-ips versions = all
20842025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
20852025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
20862025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/instances/{instance}/reboot versions = all
20872025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/instances/{instance}/serial-console versions = all
20882025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
20892025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
20902025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/instances/{instance}/start versions = all
20912025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/instances/{instance}/stop versions = all
20922025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/internet-gateway-ip-addresses versions = all
20932025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/internet-gateway-ip-addresses versions = all
20942025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
20952025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/internet-gateway-ip-pools versions = all
20962025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/internet-gateway-ip-pools versions = all
20972025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
20982025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/internet-gateways versions = all
20992025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/internet-gateways versions = all
21002025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
21012025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/internet-gateways/{gateway} versions = all
21022025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/ip-pools versions = all
21032025-03-07T22:34:54.312ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/ip-pools/{pool} versions = all
21042025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/login/{silo_name}/local versions = all
21052025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/logout versions = all
21062025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/me versions = all
21072025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/me/groups versions = all
21082025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/me/ssh-keys versions = all
21092025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/me/ssh-keys versions = all
21102025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
21112025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
21122025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/metrics/{metric_name} versions = all
21132025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/network-interfaces versions = all
21142025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/network-interfaces versions = all
21152025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/network-interfaces/{interface} versions = all
21162025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/network-interfaces/{interface} versions = all
21172025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/network-interfaces/{interface} versions = all
21182025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/ping versions = all
21192025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/policy versions = all
21202025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/policy versions = all
21212025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/projects versions = all
21222025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/projects versions = all
21232025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/projects/{project} versions = all
21242025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/projects/{project} versions = all
21252025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/projects/{project} versions = all
21262025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/projects/{project}/policy versions = all
21272025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/projects/{project}/policy versions = all
21282025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/snapshots versions = all
21292025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/snapshots versions = all
21302025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/snapshots/{snapshot} versions = all
21312025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/snapshots/{snapshot} versions = all
21322025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/disks versions = all
21332025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
21342025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
21352025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/racks versions = all
21362025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
21372025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/sleds versions = all
21382025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/hardware/sleds versions = all
21392025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
21402025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
21412025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
21422025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
21432025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
21442025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/switch-port versions = all
21452025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
21462025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
21472025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
21482025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
21492025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
21502025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/switches versions = all
21512025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
21522025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/identity-providers versions = all
21532025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/identity-providers/local/users versions = all
21542025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
21552025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
21562025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/identity-providers/saml versions = all
21572025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
21582025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/ip-pools versions = all
21592025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/ip-pools versions = all
21602025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
21612025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/ip-pools/{pool} versions = all
21622025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/system/ip-pools/{pool} versions = all
21632025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
21642025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
21652025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
21662025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
21672025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
21682025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
21692025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
21702025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
21712025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/ip-pools-service versions = all
21722025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/ip-pools-service/ranges versions = all
21732025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
21742025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
21752025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/metrics/{metric_name} versions = all
21762025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/address-lot versions = all
21772025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/networking/address-lot versions = all
21782025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
21792025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
21802025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/allow-list versions = all
21812025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/system/networking/allow-list versions = all
21822025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/networking/bfd-disable versions = all
21832025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/networking/bfd-enable versions = all
21842025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/bfd-status versions = all
21852025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/system/networking/bgp versions = all
21862025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/bgp versions = all
21872025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/networking/bgp versions = all
21882025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/bgp-announce-set versions = all
21892025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
21902025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
21912025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
21922025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/bgp-exported versions = all
21932025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/bgp-message-history versions = all
21942025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
21952025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/bgp-status versions = all
21962025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/loopback-address versions = all
21972025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/networking/loopback-address versions = all
21982025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
21992025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
22002025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/switch-port-settings versions = all
22012025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/networking/switch-port-settings versions = all
22022025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
22032025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/policy versions = all
22042025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/system/policy versions = all
22052025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/roles versions = all
22062025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/roles/{role_name} versions = all
22072025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/silo-quotas versions = all
22082025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/silos versions = all
22092025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/silos versions = all
22102025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/system/silos/{silo} versions = all
22112025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/silos/{silo} versions = all
22122025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
22132025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/silos/{silo}/policy versions = all
22142025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/system/silos/{silo}/policy versions = all
22152025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/silos/{silo}/quotas versions = all
22162025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
22172025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/system/timeseries/query versions = all
22182025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/timeseries/schemas versions = all
22192025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/system/update/repository versions = all
22202025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/update/repository/{system_version} versions = all
22212025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/update/target-release versions = all
22222025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/system/update/target-release versions = all
22232025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/users versions = all
22242025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/users/{user_id} versions = all
22252025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/users-builtin versions = all
22262025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/users-builtin/{user} versions = all
22272025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/utilization/silos versions = all
22282025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/system/utilization/silos/{silo} versions = all
22292025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/timeseries/query versions = all
22302025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/users versions = all
22312025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/utilization versions = all
22322025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/vpc-firewall-rules versions = all
22332025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/vpc-firewall-rules versions = all
22342025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/vpc-router-routes versions = all
22352025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/vpc-router-routes versions = all
22362025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
22372025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/vpc-router-routes/{route} versions = all
22382025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/vpc-router-routes/{route} versions = all
22392025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/vpc-routers versions = all
22402025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/vpc-routers versions = all
22412025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/vpc-routers/{router} versions = all
22422025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/vpc-routers/{router} versions = all
22432025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/vpc-routers/{router} versions = all
22442025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/vpc-subnets versions = all
22452025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/vpc-subnets versions = all
22462025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
22472025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/vpc-subnets/{subnet} versions = all
22482025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
22492025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
22502025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/vpcs versions = all
22512025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = POST path = /v1/vpcs versions = all
22522025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = DELETE path = /v1/vpcs/{vpc} versions = all
22532025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = GET path = /v1/vpcs/{vpc} versions = all
22542025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.6:443 method = PUT path = /v1/vpcs/{vpc} versions = all
22552025-03-07T22:34:54.313ZINFOnexus (dropshot_external): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 172.30.2.6:443
22562025-03-07T22:34:54.313ZDEBGnexus (dropshot_external): successfully registered DTrace USDT probes local_addr = 172.30.2.6:443
22572025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = / versions = all
22582025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /access versions = all
22592025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /assets/{path} versions = all
22602025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /device/auth versions = all
22612025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /device/confirm versions = all
22622025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /device/success versions = all
22632025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /device/token versions = all
22642025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /device/verify versions = all
22652025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /experimental/v1/probes versions = all
22662025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /experimental/v1/probes versions = all
22672025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /experimental/v1/probes/{probe} versions = all
22682025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /experimental/v1/probes/{probe} versions = all
22692025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /experimental/v1/system/support-bundles versions = all
22702025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /experimental/v1/system/support-bundles versions = all
22712025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
22722025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
22732025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
22742025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
22752025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
22762025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
22772025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
22782025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /images versions = all
22792025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /login versions = all
22802025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /login/{silo_name}/local versions = all
22812025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
22822025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
22832025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
22842025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /lookup/{path} versions = all
22852025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /projects/{path} versions = all
22862025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /projects-new versions = all
22872025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /settings/{path} versions = all
22882025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /system/{path} versions = all
22892025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /utilization versions = all
22902025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/affinity-groups versions = all
22912025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/affinity-groups versions = all
22922025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
22932025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
22942025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
22952025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
22962025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
22972025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
22982025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
22992025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/anti-affinity-groups versions = all
23002025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/anti-affinity-groups versions = all
23012025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
23022025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
23032025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
23042025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
23052025-03-07T22:34:54.320ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
23062025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
23072025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
23082025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/certificates versions = all
23092025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/certificates versions = all
23102025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/certificates/{certificate} versions = all
23112025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/certificates/{certificate} versions = all
23122025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/disks versions = all
23132025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/disks versions = all
23142025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/disks/{disk} versions = all
23152025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/disks/{disk} versions = all
23162025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/disks/{disk}/bulk-write versions = all
23172025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
23182025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
23192025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/disks/{disk}/finalize versions = all
23202025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
23212025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/floating-ips versions = all
23222025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/floating-ips versions = all
23232025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
23242025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/floating-ips/{floating_ip} versions = all
23252025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
23262025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
23272025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
23282025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/groups versions = all
23292025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/groups/{group_id} versions = all
23302025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/images versions = all
23312025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/images versions = all
23322025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/images/{image} versions = all
23332025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/images/{image} versions = all
23342025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/images/{image}/demote versions = all
23352025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/images/{image}/promote versions = all
23362025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/instances versions = all
23372025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/instances versions = all
23382025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/instances/{instance} versions = all
23392025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/instances/{instance} versions = all
23402025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/instances/{instance} versions = all
23412025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/instances/{instance}/disks versions = all
23422025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/instances/{instance}/disks/attach versions = all
23432025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/instances/{instance}/disks/detach versions = all
23442025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/instances/{instance}/external-ips versions = all
23452025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
23462025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
23472025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/instances/{instance}/reboot versions = all
23482025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/instances/{instance}/serial-console versions = all
23492025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
23502025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
23512025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/instances/{instance}/start versions = all
23522025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/instances/{instance}/stop versions = all
23532025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/internet-gateway-ip-addresses versions = all
23542025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/internet-gateway-ip-addresses versions = all
23552025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
23562025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/internet-gateway-ip-pools versions = all
23572025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/internet-gateway-ip-pools versions = all
23582025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
23592025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/internet-gateways versions = all
23602025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/internet-gateways versions = all
23612025-03-07T22:34:54.321ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
23622025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/internet-gateways/{gateway} versions = all
23632025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/ip-pools versions = all
23642025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/ip-pools/{pool} versions = all
23652025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/login/{silo_name}/local versions = all
23662025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/logout versions = all
23672025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/me versions = all
23682025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/me/groups versions = all
23692025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/me/ssh-keys versions = all
23702025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/me/ssh-keys versions = all
23712025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
23722025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
23732025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/metrics/{metric_name} versions = all
23742025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/network-interfaces versions = all
23752025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/network-interfaces versions = all
23762025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/network-interfaces/{interface} versions = all
23772025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/network-interfaces/{interface} versions = all
23782025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/network-interfaces/{interface} versions = all
23792025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/ping versions = all
23802025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/policy versions = all
23812025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/policy versions = all
23822025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/projects versions = all
23832025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/projects versions = all
23842025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/projects/{project} versions = all
23852025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/projects/{project} versions = all
23862025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/projects/{project} versions = all
23872025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/projects/{project}/policy versions = all
23882025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/projects/{project}/policy versions = all
23892025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/snapshots versions = all
23902025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/snapshots versions = all
23912025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/snapshots/{snapshot} versions = all
23922025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/snapshots/{snapshot} versions = all
23932025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/disks versions = all
23942025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
23952025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
23962025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/racks versions = all
23972025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
23982025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/sleds versions = all
23992025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/hardware/sleds versions = all
24002025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
24012025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
24022025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
24032025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
24042025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
24052025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/switch-port versions = all
24062025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
24072025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
24082025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
24092025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
24102025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
24112025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/switches versions = all
24122025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
24132025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/identity-providers versions = all
24142025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/identity-providers/local/users versions = all
24152025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
24162025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
24172025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/identity-providers/saml versions = all
24182025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
24192025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/ip-pools versions = all
24202025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/ip-pools versions = all
24212025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
24222025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/ip-pools/{pool} versions = all
24232025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/system/ip-pools/{pool} versions = all
24242025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
24252025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
24262025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
24272025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
24282025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
24292025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
24302025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
24312025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
24322025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/ip-pools-service versions = all
24332025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/ip-pools-service/ranges versions = all
24342025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
24352025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
24362025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/metrics/{metric_name} versions = all
24372025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/address-lot versions = all
24382025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/networking/address-lot versions = all
24392025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
24402025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
24412025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/allow-list versions = all
24422025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/system/networking/allow-list versions = all
24432025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/networking/bfd-disable versions = all
24442025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/networking/bfd-enable versions = all
24452025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/bfd-status versions = all
24462025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/system/networking/bgp versions = all
24472025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/bgp versions = all
24482025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/networking/bgp versions = all
24492025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/bgp-announce-set versions = all
24502025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
24512025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
24522025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
24532025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/bgp-exported versions = all
24542025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/bgp-message-history versions = all
24552025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
24562025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/bgp-status versions = all
24572025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/loopback-address versions = all
24582025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/networking/loopback-address versions = all
24592025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
24602025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
24612025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/switch-port-settings versions = all
24622025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/networking/switch-port-settings versions = all
24632025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
24642025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/policy versions = all
24652025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/system/policy versions = all
24662025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/roles versions = all
24672025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/roles/{role_name} versions = all
24682025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/silo-quotas versions = all
24692025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/silos versions = all
24702025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/silos versions = all
24712025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/system/silos/{silo} versions = all
24722025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/silos/{silo} versions = all
24732025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
24742025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/silos/{silo}/policy versions = all
24752025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/system/silos/{silo}/policy versions = all
24762025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/silos/{silo}/quotas versions = all
24772025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
24782025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/system/timeseries/query versions = all
24792025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/timeseries/schemas versions = all
24802025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/system/update/repository versions = all
24812025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/update/repository/{system_version} versions = all
24822025-03-07T22:34:54.322ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/update/target-release versions = all
24832025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/system/update/target-release versions = all
24842025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/users versions = all
24852025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/users/{user_id} versions = all
24862025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/users-builtin versions = all
24872025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/users-builtin/{user} versions = all
24882025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/utilization/silos versions = all
24892025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/system/utilization/silos/{silo} versions = all
24902025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/timeseries/query versions = all
24912025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/users versions = all
24922025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/utilization versions = all
24932025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/vpc-firewall-rules versions = all
24942025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/vpc-firewall-rules versions = all
24952025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/vpc-router-routes versions = all
24962025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/vpc-router-routes versions = all
24972025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
24982025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/vpc-router-routes/{route} versions = all
24992025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/vpc-router-routes/{route} versions = all
25002025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/vpc-routers versions = all
25012025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/vpc-routers versions = all
25022025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/vpc-routers/{router} versions = all
25032025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/vpc-routers/{router} versions = all
25042025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/vpc-routers/{router} versions = all
25052025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/vpc-subnets versions = all
25062025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/vpc-subnets versions = all
25072025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
25082025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/vpc-subnets/{subnet} versions = all
25092025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
25102025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
25112025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/vpcs versions = all
25122025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = POST path = /v1/vpcs versions = all
25132025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = DELETE path = /v1/vpcs/{vpc} versions = all
25142025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = GET path = /v1/vpcs/{vpc} versions = all
25152025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::b]:12228 method = PUT path = /v1/vpcs/{vpc} versions = all
25162025-03-07T22:34:54.323ZINFOnexus (dropshot_external_techport): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [fd00:1122:3344:101::b]:12228
25172025-03-07T22:34:54.323ZDEBGnexus (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::b]:12228
25182025-03-07T22:34:54.329ZDEBGnexus (nexus-producer-server): registered DTrace probes
25192025-03-07T22:34:54.339ZDEBGnexus (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::b]:53580 method = GET path = /{producer_id} versions = all
25202025-03-07T22:34:54.341ZINFOnexus (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [fd00:1122:3344:101::b]:53580
25212025-03-07T22:34:54.341ZDEBGnexus (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::b]:53580
25222025-03-07T22:34:54.341ZDEBGnexus (nexus-producer-server): Requested any available port, Dropshot server has been bound to [fd00:1122:3344:101::b]:53580
25232025-03-07T22:34:54.341ZDEBGnexus (nexus-producer-server): Nexus IP provided explicitly, will use it addr = [fd00:1122:3344:101::b]:12221
25242025-03-07T22:34:54.341ZDEBGnexus (nexus-producer-server): starting producer registration task
25252025-03-07T22:34:54.342ZINFOnexus (nexus-producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::b]:53580 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = dfe169a3-7536-4b54-9593-8815d6e81dd4
25262025-03-07T22:34:54.342ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
25272025-03-07T22:34:54.342ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
25282025-03-07T22:34:54.405ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
25292025-03-07T22:34:54.405ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e86cf5d-5098-4885-aaa8-1c8ff7d05380", "content-length": "691", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
25302025-03-07T22:34:54.405ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
25312025-03-07T22:34:54.405ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
25322025-03-07T22:34:54.405ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
25332025-03-07T22:34:54.406ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 446.926675ms iteration = 3 status = {}
25342025-03-07T22:34:54.409ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
25352025-03-07T22:34:54.420ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
25362025-03-07T22:34:54.420ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "c08aa51a-de3d-448a-801e-a2eb7b1ac8ba", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
25372025-03-07T22:34:54.420ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
25382025-03-07T22:34:54.422ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "f3fb44c1-1edc-41d9-a0b5-dcc806c6baba", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
25392025-03-07T22:34:54.422ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
25402025-03-07T22:34:54.424ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
25412025-03-07T22:34:54.424ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 233.419883ms iteration = 4 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
25422025-03-07T22:34:54.466ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
25432025-03-07T22:34:54.466ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:47017
25442025-03-07T22:34:54.488ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:47017 req_id = 622c1954-e079-4eed-b3b6-1eef18845a3c roles = RoleSet { roles: {} } uri = /metrics/producers
25452025-03-07T22:34:54.488ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:47017 req_id = 622c1954-e079-4eed-b3b6-1eef18845a3c resource = Database result = Ok(()) uri = /metrics/producers
25462025-03-07T22:34:54.497ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25472025-03-07T22:34:54.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25482025-03-07T22:34:54.504ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
25492025-03-07T22:34:54.504ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25502025-03-07T22:34:54.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25512025-03-07T22:34:54.518ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
25522025-03-07T22:34:54.519ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "b585ac57-7133-4746-82fc-09463e6be3ad", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25532025-03-07T22:34:54.519ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25542025-03-07T22:34:54.520ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "4bcb3b70-f3fb-4999-ab2b-5e4fc080c401", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25552025-03-07T22:34:54.520ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25562025-03-07T22:34:54.522ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "7201e019-0285-4cd1-8eed-8260b9f4ba29", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25572025-03-07T22:34:54.522ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25582025-03-07T22:34:54.524ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25592025-03-07T22:34:54.524ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 335.615506ms iteration = 4 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
25602025-03-07T22:34:54.526ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
25612025-03-07T22:34:54.526ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e05137a-abf5-448b-a7cf-8ce2ee236ca5", "content-length": "134", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
25622025-03-07T22:34:54.526ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
25632025-03-07T22:34:54.526ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
25642025-03-07T22:34:54.526ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
25652025-03-07T22:34:54.526ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
25662025-03-07T22:34:54.526ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25672025-03-07T22:34:54.526ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25682025-03-07T22:34:54.526ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25692025-03-07T22:34:54.526ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
25702025-03-07T22:34:54.527ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25712025-03-07T22:34:54.527ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
25722025-03-07T22:34:54.527ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
25732025-03-07T22:34:54.527ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25742025-03-07T22:34:54.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25752025-03-07T22:34:54.533ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
25762025-03-07T22:34:54.533ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4ce419f-af82-4766-bf0b-970d83618040", "content-length": "2", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
25772025-03-07T22:34:54.533ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
25782025-03-07T22:34:54.534ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "549362ab-a9f5-43a5-a969-76ccf18a9073", "content-length": "52", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
25792025-03-07T22:34:54.534ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
25802025-03-07T22:34:54.534ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
25812025-03-07T22:34:54.575ZINFOnexus: assigned collector to new producer collector_id = 7b5e9a24-6b39-4a58-a275-8cc5a75195d8 file = nexus/src/app/oximeter.rs:76 producer_id = dfe169a3-7536-4b54-9593-8815d6e81dd4
25822025-03-07T22:34:54.575ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 108907 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:47017 req_id = 622c1954-e079-4eed-b3b6-1eef18845a3c response_code = 201 uri = /metrics/producers
25832025-03-07T22:34:54.575ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "622c1954-e079-4eed-b3b6-1eef18845a3c", "content-length": "41", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
25842025-03-07T22:34:54.578ZDEBGnexus (producer-registration-task): registered with nexus successfully
25852025-03-07T22:34:54.578ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
25862025-03-07T22:34:54.658ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "2f97c7c1-be43-405d-80c6-813374367eff", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
25872025-03-07T22:34:54.658ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
25882025-03-07T22:34:54.658ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
25892025-03-07T22:34:54.658ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "00e73cda-d209-4837-9324-949a1ce63f06", "content-length": "682", "date": "Fri, 07 Mar 2025 22:34:54 GMT"} })
25902025-03-07T22:34:54.658ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25912025-03-07T22:34:54.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25922025-03-07T22:34:54.662ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25932025-03-07T22:34:54.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25942025-03-07T22:34:54.667ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25952025-03-07T22:34:54.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25962025-03-07T22:34:54.677ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25972025-03-07T22:34:54.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25982025-03-07T22:34:54.705ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 745.532884ms iteration = 3 status = {}
25992025-03-07T22:35:03.959ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 8 reason = Timeout
26002025-03-07T22:35:03.959ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 8 reason = Timeout
26012025-03-07T22:35:03.959ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26022025-03-07T22:35:03.959ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26032025-03-07T22:35:03.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26042025-03-07T22:35:03.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26052025-03-07T22:35:03.962ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26062025-03-07T22:35:03.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26072025-03-07T22:35:03.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26082025-03-07T22:35:03.964ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.21748ms iteration = 8 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
26092025-03-07T22:35:03.964ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
26102025-03-07T22:35:03.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26112025-03-07T22:35:03.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26122025-03-07T22:35:03.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26132025-03-07T22:35:03.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26142025-03-07T22:35:03.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26152025-03-07T22:35:03.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
26162025-03-07T22:35:03.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26172025-03-07T22:35:03.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26182025-03-07T22:35:03.981ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
26192025-03-07T22:35:03.981ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.416365ms iteration = 8 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
26202025-03-07T22:35:04.661ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56191 req_id = 9b602fc9-4044-4b26-9a50-785ec5bd3f05 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
26212025-03-07T22:35:04.661ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56191 req_id = 9b602fc9-4044-4b26-9a50-785ec5bd3f05 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
26222025-03-07T22:35:04.664ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3219 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56191 req_id = 9b602fc9-4044-4b26-9a50-785ec5bd3f05 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
26232025-03-07T22:35:04.665ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56191 req_id = 96e6d3d1-f03a-48d2-85e9-1a55e4f7acee roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
26242025-03-07T22:35:04.665ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56191 req_id = 96e6d3d1-f03a-48d2-85e9-1a55e4f7acee resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
26252025-03-07T22:35:04.667ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2270 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56191 req_id = 96e6d3d1-f03a-48d2-85e9-1a55e4f7acee response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
26262025-03-07T22:35:13.960ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 9 reason = Timeout
26272025-03-07T22:35:13.960ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 9 reason = Timeout
26282025-03-07T22:35:13.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26292025-03-07T22:35:13.960ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26302025-03-07T22:35:13.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26312025-03-07T22:35:13.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26322025-03-07T22:35:13.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26332025-03-07T22:35:13.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26342025-03-07T22:35:13.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26352025-03-07T22:35:13.965ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
26362025-03-07T22:35:13.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26372025-03-07T22:35:13.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26382025-03-07T22:35:13.965ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.970488ms iteration = 9 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
26392025-03-07T22:35:13.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26402025-03-07T22:35:13.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26412025-03-07T22:35:13.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26422025-03-07T22:35:13.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
26432025-03-07T22:35:13.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26442025-03-07T22:35:13.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26452025-03-07T22:35:13.981ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
26462025-03-07T22:35:13.981ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.509737ms iteration = 9 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
26472025-03-07T22:35:14.670ZINFOnexus (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:53580 remote_addr = [fd00:1122:3344:101::d]:42651
26482025-03-07T22:35:14.683ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 12722 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 973b66c4-ff2b-45a4-926c-93c5a2fa93e2 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
26492025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 10 reason = Timeout
26502025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26512025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 10 reason = Timeout
26522025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 4 reason = Timeout
26532025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
26542025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
26552025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26562025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 5 reason = Timeout
26572025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26582025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 4 reason = Timeout
26592025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 4 reason = Timeout
26602025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
26612025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
26622025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 4 reason = Timeout
26632025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 4 reason = Timeout
26642025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 4 reason = Timeout
26652025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 4 reason = Timeout
26662025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26672025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
26682025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
26692025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 4 reason = Timeout
26702025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26712025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 4 reason = Timeout
26722025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
26732025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 4 reason = Timeout
26742025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26752025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 4 reason = Timeout
26762025-03-07T22:35:23.960ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
26772025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26782025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
26792025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26802025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26812025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26822025-03-07T22:35:23.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
26832025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26842025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26852025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
26862025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26872025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26882025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
26892025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
26902025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
26912025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 4 reason = Timeout
26922025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 4 reason = Timeout
26932025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
26942025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
26952025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 4 reason = Timeout
26962025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
26972025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 4 reason = Timeout
26982025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
26992025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 4 reason = Timeout
27002025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27012025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
27022025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
27032025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27042025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
27052025-03-07T22:35:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
27062025-03-07T22:35:23.964ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27072025-03-07T22:35:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27082025-03-07T22:35:23.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27092025-03-07T22:35:23.964ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27102025-03-07T22:35:23.966ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.682664ms iteration = 4 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
27112025-03-07T22:35:23.966ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27122025-03-07T22:35:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
27132025-03-07T22:35:23.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27142025-03-07T22:35:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27152025-03-07T22:35:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
27162025-03-07T22:35:23.968ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27172025-03-07T22:35:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27182025-03-07T22:35:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
27192025-03-07T22:35:23.969ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27202025-03-07T22:35:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27212025-03-07T22:35:23.969ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27222025-03-07T22:35:23.970ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 882, tv_nsec: 327527046 } })
27232025-03-07T22:35:23.970ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27242025-03-07T22:35:23.970ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
27252025-03-07T22:35:23.970ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27262025-03-07T22:35:23.971ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27272025-03-07T22:35:23.971ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 882, tv_nsec: 329219023 } })
27282025-03-07T22:35:23.971ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
27292025-03-07T22:35:23.972ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
27302025-03-07T22:35:23.972ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27312025-03-07T22:35:23.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27322025-03-07T22:35:23.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
27332025-03-07T22:35:23.972ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27342025-03-07T22:35:23.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27352025-03-07T22:35:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
27362025-03-07T22:35:23.973ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27372025-03-07T22:35:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27382025-03-07T22:35:23.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
27392025-03-07T22:35:23.974ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
27402025-03-07T22:35:23.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
27412025-03-07T22:35:23.977ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
27422025-03-07T22:35:23.977ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 16.820082ms iteration = 10 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
27432025-03-07T22:35:23.977ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 15.63711ms iteration = 4 status = {"errors":[],"finish_invoked_ok":[]}
27442025-03-07T22:35:23.977ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
27452025-03-07T22:35:23.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
27462025-03-07T22:35:23.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
27472025-03-07T22:35:23.977ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27482025-03-07T22:35:23.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27492025-03-07T22:35:23.978ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27502025-03-07T22:35:23.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27512025-03-07T22:35:23.980ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 882, tv_nsec: 337939257 } })
27522025-03-07T22:35:23.980ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
27532025-03-07T22:35:23.981ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 19.476037ms iteration = 4 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
27542025-03-07T22:35:23.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27552025-03-07T22:35:23.982ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27562025-03-07T22:35:23.982ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27572025-03-07T22:35:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27582025-03-07T22:35:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
27592025-03-07T22:35:23.984ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27602025-03-07T22:35:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
27612025-03-07T22:35:23.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27622025-03-07T22:35:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27632025-03-07T22:35:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27642025-03-07T22:35:23.996ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
27652025-03-07T22:35:23.996ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 35.995776ms iteration = 10 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
27662025-03-07T22:35:24.000ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
27672025-03-07T22:35:24.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
27682025-03-07T22:35:24.015ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27692025-03-07T22:35:24.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27702025-03-07T22:35:24.016ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 54.639025ms iteration = 4 status = {"errors":[],"garbage_collect_requested":[]}
27712025-03-07T22:35:24.017ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27722025-03-07T22:35:24.017ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27732025-03-07T22:35:24.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27742025-03-07T22:35:24.018ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
27752025-03-07T22:35:24.018ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27762025-03-07T22:35:24.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27772025-03-07T22:35:24.019ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 57.877436ms iteration = 4 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
27782025-03-07T22:35:24.020ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27792025-03-07T22:35:24.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27802025-03-07T22:35:24.022ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27812025-03-07T22:35:24.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
27822025-03-07T22:35:24.024ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27832025-03-07T22:35:24.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27842025-03-07T22:35:24.026ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 64.51976ms iteration = 4 status = {"errors":[],"requests_created_ok":[]}
27852025-03-07T22:35:24.029ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
27862025-03-07T22:35:24.029ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 68.735801ms iteration = 4 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
27872025-03-07T22:35:24.030ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 70.340016ms iteration = 4 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
27882025-03-07T22:35:24.031ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
27892025-03-07T22:35:24.031ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 71.043152ms iteration = 4 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
27902025-03-07T22:35:24.074ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
27912025-03-07T22:35:24.074ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
27922025-03-07T22:35:24.074ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
27932025-03-07T22:35:24.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
27942025-03-07T22:35:24.082ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
27952025-03-07T22:35:24.082ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
27962025-03-07T22:35:24.083ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c980f648-8ff1-4705-86d8-e46b525d29fd", "content-length": "26", "date": "Fri, 07 Mar 2025 22:35:24 GMT"} })
27972025-03-07T22:35:24.083ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
27982025-03-07T22:35:24.085ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
27992025-03-07T22:35:24.102ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28002025-03-07T22:35:24.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28012025-03-07T22:35:24.104ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28022025-03-07T22:35:24.105ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
28032025-03-07T22:35:24.105ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28042025-03-07T22:35:24.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28052025-03-07T22:35:24.107ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28062025-03-07T22:35:24.110ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
28072025-03-07T22:35:24.110ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28082025-03-07T22:35:24.110ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28092025-03-07T22:35:24.111ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28102025-03-07T22:35:24.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28112025-03-07T22:35:24.113ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
28122025-03-07T22:35:24.113ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
28132025-03-07T22:35:24.113ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
28142025-03-07T22:35:24.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
28152025-03-07T22:35:24.114ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28162025-03-07T22:35:24.114ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28172025-03-07T22:35:24.116ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28182025-03-07T22:35:24.117ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 156.846267ms iteration = 5 status = {"physical_disks_added":0}
28192025-03-07T22:35:24.118ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
28202025-03-07T22:35:24.118ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28212025-03-07T22:35:24.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28222025-03-07T22:35:24.129ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 168.60529ms iteration = 4 status = 0
28232025-03-07T22:35:24.147ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
28242025-03-07T22:35:24.147ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
28252025-03-07T22:35:24.147ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 187.38632ms iteration = 4 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
28262025-03-07T22:35:24.155ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
28272025-03-07T22:35:24.155ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
28282025-03-07T22:35:24.156ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "97543246-5b0e-41c6-83c7-4898b05317ac", "content-length": "26", "date": "Fri, 07 Mar 2025 22:35:24 GMT"} })
28292025-03-07T22:35:24.156ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
28302025-03-07T22:35:24.157ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
28312025-03-07T22:35:24.171ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
28322025-03-07T22:35:24.171ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
28332025-03-07T22:35:24.172ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "08a843e6-030a-4c9f-9b88-6ec655228a94", "content-length": "26", "date": "Fri, 07 Mar 2025 22:35:24 GMT"} })
28342025-03-07T22:35:24.172ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
28352025-03-07T22:35:24.173ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
28362025-03-07T22:35:24.236ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
28372025-03-07T22:35:24.236ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
28382025-03-07T22:35:24.237ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "92fef545-fafb-4557-9cbe-c5aa08c36edc", "content-length": "372", "date": "Fri, 07 Mar 2025 22:35:24 GMT"} })
28392025-03-07T22:35:24.237ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28402025-03-07T22:35:24.237ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28412025-03-07T22:35:24.255ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
28422025-03-07T22:35:24.255ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
28432025-03-07T22:35:24.256ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "0ca2dd24-7dc0-4955-a36d-705991334adf", "date": "Fri, 07 Mar 2025 22:35:24 GMT"} })
28442025-03-07T22:35:24.256ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28452025-03-07T22:35:24.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28462025-03-07T22:35:24.259ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28472025-03-07T22:35:24.259ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28482025-03-07T22:35:24.266ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
28492025-03-07T22:35:24.266ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "29138ff2-8ad4-4457-a2fb-d00f6c92f969", "content-length": "1", "date": "Fri, 07 Mar 2025 22:35:24 GMT"} })
28502025-03-07T22:35:24.268ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
28512025-03-07T22:35:24.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
28522025-03-07T22:35:24.278ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 318.282729ms iteration = 4 status = {}
28532025-03-07T22:35:24.299ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
28542025-03-07T22:35:24.299ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "f21dac0f-5ccc-4f4f-ba92-dd3ab4911d30", "content-length": "691", "date": "Fri, 07 Mar 2025 22:35:24 GMT"} })
28552025-03-07T22:35:24.299ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
28562025-03-07T22:35:24.299ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
28572025-03-07T22:35:24.299ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
28582025-03-07T22:35:24.300ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 340.445682ms iteration = 4 status = null
28592025-03-07T22:35:24.301ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 340.525002ms iteration = 4 status = {}
28602025-03-07T22:35:24.329ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28612025-03-07T22:35:24.329ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28622025-03-07T22:35:24.331ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
28632025-03-07T22:35:24.331ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28642025-03-07T22:35:24.331ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28652025-03-07T22:35:24.346ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
28662025-03-07T22:35:24.347ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "05389a27-4a8e-459d-ac71-8d12515eee01", "content-length": "134", "date": "Fri, 07 Mar 2025 22:35:24 GMT"} })
28672025-03-07T22:35:24.347ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
28682025-03-07T22:35:24.347ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
28692025-03-07T22:35:24.347ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
28702025-03-07T22:35:24.347ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
28712025-03-07T22:35:24.347ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
28722025-03-07T22:35:24.347ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
28732025-03-07T22:35:24.347ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
28742025-03-07T22:35:24.347ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
28752025-03-07T22:35:24.347ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
28762025-03-07T22:35:24.347ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
28772025-03-07T22:35:24.347ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
28782025-03-07T22:35:24.347ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28792025-03-07T22:35:24.348ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28802025-03-07T22:35:24.368ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28812025-03-07T22:35:24.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28822025-03-07T22:35:24.373ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
28832025-03-07T22:35:24.373ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
28842025-03-07T22:35:24.373ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "01a4ca06-dd03-43b4-96b0-967b9d1e3962", "date": "Fri, 07 Mar 2025 22:35:24 GMT"} })
28852025-03-07T22:35:24.374ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 413.974798ms iteration = 4 status = {}
28862025-03-07T22:35:24.381ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
28872025-03-07T22:35:24.381ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "2050af89-3e8c-4d37-ad03-fe92a8026630", "content-length": "2", "date": "Fri, 07 Mar 2025 22:35:24 GMT"} })
28882025-03-07T22:35:24.381ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
28892025-03-07T22:35:24.381ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1454445-b439-49a3-a22e-f60a83bd2e66", "content-length": "52", "date": "Fri, 07 Mar 2025 22:35:24 GMT"} })
28902025-03-07T22:35:24.381ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
28912025-03-07T22:35:24.381ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
28922025-03-07T22:35:24.616ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "f0a97d8c-996d-4525-9bcb-36cb3a29a470", "date": "Fri, 07 Mar 2025 22:35:24 GMT"} })
28932025-03-07T22:35:24.616ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
28942025-03-07T22:35:24.616ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
28952025-03-07T22:35:24.616ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c6c55ac-9642-4aff-a241-942bb8e412ab", "content-length": "682", "date": "Fri, 07 Mar 2025 22:35:24 GMT"} })
28962025-03-07T22:35:24.616ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28972025-03-07T22:35:24.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28982025-03-07T22:35:24.620ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28992025-03-07T22:35:24.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29002025-03-07T22:35:24.624ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29012025-03-07T22:35:24.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29022025-03-07T22:35:24.627ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29032025-03-07T22:35:24.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29042025-03-07T22:35:24.670ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 294 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 1be03756-655c-4e36-aeb7-487854ce241b response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
29052025-03-07T22:35:24.722ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 762.240955ms iteration = 4 status = {}
29062025-03-07T22:35:33.960ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 11 reason = Timeout
29072025-03-07T22:35:33.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29082025-03-07T22:35:33.960ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 11 reason = Timeout
29092025-03-07T22:35:33.960ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29102025-03-07T22:35:33.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29112025-03-07T22:35:33.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29122025-03-07T22:35:33.962ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29132025-03-07T22:35:33.963ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29142025-03-07T22:35:33.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29152025-03-07T22:35:33.965ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
29162025-03-07T22:35:33.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29172025-03-07T22:35:33.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29182025-03-07T22:35:33.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29192025-03-07T22:35:33.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29202025-03-07T22:35:33.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29212025-03-07T22:35:33.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
29222025-03-07T22:35:33.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29232025-03-07T22:35:33.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29242025-03-07T22:35:33.983ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
29252025-03-07T22:35:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.202923ms iteration = 11 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
29262025-03-07T22:35:33.994ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.27528ms iteration = 11 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
29272025-03-07T22:35:34.671ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 274 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = e2c403a4-7004-4942-a874-7d2edb61caa0 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
29282025-03-07T22:35:43.960ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 12 reason = Timeout
29292025-03-07T22:35:43.960ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 12 reason = Timeout
29302025-03-07T22:35:43.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29312025-03-07T22:35:43.960ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29322025-03-07T22:35:43.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29332025-03-07T22:35:43.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29342025-03-07T22:35:43.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29352025-03-07T22:35:43.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29362025-03-07T22:35:43.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29372025-03-07T22:35:43.964ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.17322ms iteration = 12 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
29382025-03-07T22:35:43.965ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
29392025-03-07T22:35:43.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29402025-03-07T22:35:43.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29412025-03-07T22:35:43.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29422025-03-07T22:35:43.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29432025-03-07T22:35:43.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29442025-03-07T22:35:43.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
29452025-03-07T22:35:43.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29462025-03-07T22:35:43.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29472025-03-07T22:35:43.979ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
29482025-03-07T22:35:43.979ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.64335ms iteration = 12 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
29492025-03-07T22:35:44.671ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 300 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 00f6e585-8231-421d-9b13-10538b1cbbb9 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
29502025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 13 reason = Timeout
29512025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 6 reason = Timeout
29522025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
29532025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29542025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 3 reason = Timeout
29552025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 3 reason = Timeout
29562025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 5 reason = Timeout
29572025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
29582025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
29592025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 5 reason = Timeout
29602025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
29612025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 3 reason = Timeout
29622025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
29632025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 3 reason = Timeout
29642025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
29652025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 4 reason = Timeout
29662025-03-07T22:35:53.960ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
29672025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.99\u{b5}s iteration = 4 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
29682025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 3 reason = Timeout
29692025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
29702025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 5 reason = Timeout
29712025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
29722025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 5 reason = Timeout
29732025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
29742025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 5 reason = Timeout
29752025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
29762025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 5 reason = Timeout
29772025-03-07T22:35:53.960ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
29782025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 5 reason = Timeout
29792025-03-07T22:35:53.960ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 5 reason = Timeout
29802025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29812025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29822025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29832025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 5 reason = Timeout
29842025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 3 reason = Timeout
29852025-03-07T22:35:53.961ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:25:53.960873425 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
29862025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29872025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 13 reason = Timeout
29882025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
29892025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29902025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 5 reason = Timeout
29912025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
29922025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
29932025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 5 reason = Timeout
29942025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 5 reason = Timeout
29952025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
29962025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 3 reason = Timeout
29972025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 5 reason = Timeout
29982025-03-07T22:35:53.961ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
29992025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
30002025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 3 reason = Timeout
30012025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
30022025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30032025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30042025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
30052025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
30062025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 5 reason = Timeout
30072025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
30082025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
30092025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 3 reason = Timeout
30102025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 5 reason = Timeout
30112025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
30122025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
30132025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
30142025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 3 reason = Timeout
30152025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 5 reason = Timeout
30162025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
30172025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
30182025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 5 reason = Timeout
30192025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
30202025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 5 reason = Timeout
30212025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
30222025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30232025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
30242025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
30252025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
30262025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30272025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30282025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
30292025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
30302025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30312025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
30322025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
30332025-03-07T22:35:53.961ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 912, tv_nsec: 318652821 } })
30342025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
30352025-03-07T22:35:53.961ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 912, tv_nsec: 318671941 } })
30362025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
30372025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
30382025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
30392025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
30402025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
30412025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
30422025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
30432025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
30442025-03-07T22:35:53.961ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
30452025-03-07T22:35:53.962ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.228262ms iteration = 3 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
30462025-03-07T22:35:53.962ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
30472025-03-07T22:35:53.962ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.417554ms iteration = 3 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
30482025-03-07T22:35:53.963ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
30492025-03-07T22:35:53.963ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30502025-03-07T22:35:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
30512025-03-07T22:35:53.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30522025-03-07T22:35:53.964ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
30532025-03-07T22:35:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
30542025-03-07T22:35:53.964ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30552025-03-07T22:35:53.965ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.503793ms iteration = 5 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
30562025-03-07T22:35:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
30572025-03-07T22:35:53.965ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30582025-03-07T22:35:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30592025-03-07T22:35:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
30602025-03-07T22:35:53.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30612025-03-07T22:35:53.967ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 912, tv_nsec: 324642029 } })
30622025-03-07T22:35:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30632025-03-07T22:35:53.967ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
30642025-03-07T22:35:53.967ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
30652025-03-07T22:35:53.967ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
30662025-03-07T22:35:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
30672025-03-07T22:35:53.969ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30682025-03-07T22:35:53.970ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30692025-03-07T22:35:53.971ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:25:53.960873425 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
30702025-03-07T22:35:53.971ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 10.277619ms iteration = 3 status = {"expiration":"2025-03-07T22:25:53.960873425Z","pruned":[]}
30712025-03-07T22:35:53.971ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30722025-03-07T22:35:53.971ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
30732025-03-07T22:35:53.971ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30742025-03-07T22:35:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30752025-03-07T22:35:53.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
30762025-03-07T22:35:53.973ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30772025-03-07T22:35:53.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30782025-03-07T22:35:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
30792025-03-07T22:35:53.974ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30802025-03-07T22:35:53.974ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
30812025-03-07T22:35:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30822025-03-07T22:35:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30832025-03-07T22:35:53.975ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 912, tv_nsec: 332666635 } })
30842025-03-07T22:35:53.975ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
30852025-03-07T22:35:53.976ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30862025-03-07T22:35:53.977ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 15.710051ms iteration = 3 status = {"errors":[],"found_port_ok":[]}
30872025-03-07T22:35:53.977ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
30882025-03-07T22:35:53.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
30892025-03-07T22:35:53.978ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
30902025-03-07T22:35:53.978ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
30912025-03-07T22:35:53.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
30922025-03-07T22:35:53.979ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 18.057293ms iteration = 5 status = {"errors":[],"requests_created_ok":[]}
30932025-03-07T22:35:53.981ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 19.887201ms iteration = 5 status = {"errors":[],"finish_invoked_ok":[]}
30942025-03-07T22:35:53.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30952025-03-07T22:35:53.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30962025-03-07T22:35:53.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30972025-03-07T22:35:53.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30982025-03-07T22:35:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30992025-03-07T22:35:53.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
31002025-03-07T22:35:53.984ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
31012025-03-07T22:35:53.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
31022025-03-07T22:35:53.986ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
31032025-03-07T22:35:53.986ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
31042025-03-07T22:35:53.989ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31052025-03-07T22:35:53.989ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
31062025-03-07T22:35:53.989ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
31072025-03-07T22:35:53.989ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
31082025-03-07T22:35:53.989ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
31092025-03-07T22:35:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31102025-03-07T22:35:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
31112025-03-07T22:35:53.992ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31122025-03-07T22:35:53.992ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31132025-03-07T22:35:53.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31142025-03-07T22:35:53.993ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 912, tv_nsec: 350267935 } })
31152025-03-07T22:35:53.993ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
31162025-03-07T22:35:53.995ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
31172025-03-07T22:35:53.995ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31182025-03-07T22:35:53.995ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31192025-03-07T22:35:53.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31202025-03-07T22:35:53.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
31212025-03-07T22:35:53.995ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
31222025-03-07T22:35:53.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
31232025-03-07T22:35:53.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
31242025-03-07T22:35:53.996ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31252025-03-07T22:35:53.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31262025-03-07T22:35:53.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
31272025-03-07T22:35:53.998ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
31282025-03-07T22:35:53.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
31292025-03-07T22:35:54.001ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31302025-03-07T22:35:54.004ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
31312025-03-07T22:35:54.004ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
31322025-03-07T22:35:54.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
31332025-03-07T22:35:54.010ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
31342025-03-07T22:35:54.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
31352025-03-07T22:35:54.025ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31362025-03-07T22:35:54.025ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31372025-03-07T22:35:54.025ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
31382025-03-07T22:35:54.025ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
31392025-03-07T22:35:54.025ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
31402025-03-07T22:35:54.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31412025-03-07T22:35:54.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
31422025-03-07T22:35:54.027ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
31432025-03-07T22:35:54.027ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
31442025-03-07T22:35:54.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
31452025-03-07T22:35:54.028ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
31462025-03-07T22:35:54.028ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
31472025-03-07T22:35:54.028ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 66.71939ms iteration = 3 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
31482025-03-07T22:35:54.028ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
31492025-03-07T22:35:54.028ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
31502025-03-07T22:35:54.028ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
31512025-03-07T22:35:54.028ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
31522025-03-07T22:35:54.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
31532025-03-07T22:35:54.033ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31542025-03-07T22:35:54.034ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
31552025-03-07T22:35:54.034ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
31562025-03-07T22:35:54.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
31572025-03-07T22:35:54.036ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
31582025-03-07T22:35:54.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
31592025-03-07T22:35:54.036ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 75.335623ms iteration = 5 status = {"errors":[],"garbage_collect_requested":[]}
31602025-03-07T22:35:54.040ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
31612025-03-07T22:35:54.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
31622025-03-07T22:35:54.040ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 79.951278ms iteration = 13 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
31632025-03-07T22:35:54.045ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
31642025-03-07T22:35:54.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
31652025-03-07T22:35:54.062ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
31662025-03-07T22:35:54.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
31672025-03-07T22:35:54.081ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
31682025-03-07T22:35:54.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
31692025-03-07T22:35:54.090ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
31702025-03-07T22:35:54.117ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
31712025-03-07T22:35:54.127ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
31722025-03-07T22:35:54.127ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
31732025-03-07T22:35:54.127ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "64f6eb60-e2f2-4537-a190-18e708c8dc13", "content-length": "26", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} })
31742025-03-07T22:35:54.127ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
31752025-03-07T22:35:54.129ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
31762025-03-07T22:35:54.146ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31772025-03-07T22:35:54.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31782025-03-07T22:35:54.160ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
31792025-03-07T22:35:54.160ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
31802025-03-07T22:35:54.160ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f72d8e81-712c-4910-9ba2-24302506cdd0", "content-length": "26", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} })
31812025-03-07T22:35:54.160ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
31822025-03-07T22:35:54.161ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
31832025-03-07T22:35:54.163ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
31842025-03-07T22:35:54.164ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
31852025-03-07T22:35:54.164ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "175786cd-10e7-4794-93b2-f6d33c5c4560", "content-length": "26", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} })
31862025-03-07T22:35:54.164ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
31872025-03-07T22:35:54.165ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
31882025-03-07T22:35:54.167ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
31892025-03-07T22:35:54.167ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
31902025-03-07T22:35:54.167ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "71980d8f-a67e-454b-b5f0-50f71726384f", "content-length": "372", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} })
31912025-03-07T22:35:54.167ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31922025-03-07T22:35:54.167ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 207.09442ms iteration = 6 status = {"physical_disks_added":0}
31932025-03-07T22:35:54.167ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31942025-03-07T22:35:54.172ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31952025-03-07T22:35:54.173ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 212.059027ms iteration = 5 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
31962025-03-07T22:35:54.174ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31972025-03-07T22:35:54.176ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
31982025-03-07T22:35:54.176ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
31992025-03-07T22:35:54.176ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
32002025-03-07T22:35:54.182ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
32012025-03-07T22:35:54.182ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
32022025-03-07T22:35:54.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
32032025-03-07T22:35:54.184ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32042025-03-07T22:35:54.184ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32052025-03-07T22:35:54.193ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
32062025-03-07T22:35:54.193ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 232.949948ms iteration = 5 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
32072025-03-07T22:35:54.213ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 252.244104ms iteration = 5 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
32082025-03-07T22:35:54.216ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32092025-03-07T22:35:54.220ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
32102025-03-07T22:35:54.220ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 259.968148ms iteration = 13 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
32112025-03-07T22:35:54.220ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
32122025-03-07T22:35:54.220ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
32132025-03-07T22:35:54.220ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 259.880158ms iteration = 3 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
32142025-03-07T22:35:54.221ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
32152025-03-07T22:35:54.221ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
32162025-03-07T22:35:54.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
32172025-03-07T22:35:54.224ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
32182025-03-07T22:35:54.224ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 263.24722ms iteration = 5 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
32192025-03-07T22:35:54.230ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
32202025-03-07T22:35:54.230ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
32212025-03-07T22:35:54.230ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
32222025-03-07T22:35:54.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
32232025-03-07T22:35:54.230ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 270.179687ms iteration = 5 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
32242025-03-07T22:35:54.232ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32252025-03-07T22:35:54.235ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
32262025-03-07T22:35:54.235ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
32272025-03-07T22:35:54.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
32282025-03-07T22:35:54.240ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32292025-03-07T22:35:54.243ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
32302025-03-07T22:35:54.243ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32312025-03-07T22:35:54.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32322025-03-07T22:35:54.248ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 288.079748ms iteration = 5 status = 0
32332025-03-07T22:35:54.256ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
32342025-03-07T22:35:54.256ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
32352025-03-07T22:35:54.256ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "73718ad7-6ae2-4cf6-b127-52f64c8c3fc3", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} })
32362025-03-07T22:35:54.256ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32372025-03-07T22:35:54.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32382025-03-07T22:35:54.259ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32392025-03-07T22:35:54.260ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32402025-03-07T22:35:54.263ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32412025-03-07T22:35:54.263ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32422025-03-07T22:35:54.266ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
32432025-03-07T22:35:54.266ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
32442025-03-07T22:35:54.267ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "d083d5d0-e021-4a59-99f0-ff25248288f5", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} })
32452025-03-07T22:35:54.268ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
32462025-03-07T22:35:54.268ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "03f7e7ad-f171-44e6-909c-f0f51c254466", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
32472025-03-07T22:35:54.268ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
32482025-03-07T22:35:54.269ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 308.428094ms iteration = 5 status = {}
32492025-03-07T22:35:54.270ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) }
32502025-03-07T22:35:54.270ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 309.609095ms iteration = 3 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
32512025-03-07T22:35:54.270ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "6dd525ea-36f4-4f8a-9e30-e89b6a76c50d", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
32522025-03-07T22:35:54.270ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
32532025-03-07T22:35:54.272ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
32542025-03-07T22:35:54.272ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 311.741526ms iteration = 5 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
32552025-03-07T22:35:54.275ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
32562025-03-07T22:35:54.275ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
32572025-03-07T22:35:54.275ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
32582025-03-07T22:35:54.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
32592025-03-07T22:35:54.278ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32602025-03-07T22:35:54.281ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
32612025-03-07T22:35:54.282ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
32622025-03-07T22:35:54.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
32632025-03-07T22:35:54.289ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
32642025-03-07T22:35:54.301ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 341.221838ms iteration = 5 status = {}
32652025-03-07T22:35:54.317ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32662025-03-07T22:35:54.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32672025-03-07T22:35:54.325ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
32682025-03-07T22:35:54.325ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
32692025-03-07T22:35:54.325ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
32702025-03-07T22:35:54.325ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
32712025-03-07T22:35:54.327ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
32722025-03-07T22:35:54.327ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 366.111898ms iteration = 3 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
32732025-03-07T22:35:54.338ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
32742025-03-07T22:35:54.338ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
32752025-03-07T22:35:54.338ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-07 22:34:17.208516 UTC dns_group = internal generation = 1
32762025-03-07T22:35:54.338ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 377.911471ms iteration = 3 status = {"generation":1}
32772025-03-07T22:35:54.349ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32782025-03-07T22:35:54.351ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
32792025-03-07T22:35:54.351ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32802025-03-07T22:35:54.351ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32812025-03-07T22:35:54.353ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32822025-03-07T22:35:54.354ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
32832025-03-07T22:35:54.354ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32842025-03-07T22:35:54.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32852025-03-07T22:35:54.358ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
32862025-03-07T22:35:54.358ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa313297-fb8b-4a84-b2ae-b0065df5586a", "content-length": "1", "date": "Fri, 07 Mar 2025 22:35:53 GMT"} })
32872025-03-07T22:35:54.360ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
32882025-03-07T22:35:54.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
32892025-03-07T22:35:54.362ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 401.983893ms iteration = 5 status = null
32902025-03-07T22:35:54.367ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
32912025-03-07T22:35:54.367ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
32922025-03-07T22:35:54.367ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 406.398266ms iteration = 5 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
32932025-03-07T22:35:54.374ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
32942025-03-07T22:35:54.375ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "acba896f-ebe3-43f5-8d52-4470c8823371", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
32952025-03-07T22:35:54.375ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
32962025-03-07T22:35:54.376ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "20610e4b-38a3-4476-8bf0-06262ecdbbf3", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
32972025-03-07T22:35:54.376ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
32982025-03-07T22:35:54.377ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "e36c3db8-a92f-45f3-b601-a95a22887093", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
32992025-03-07T22:35:54.377ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
33002025-03-07T22:35:54.379ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
33012025-03-07T22:35:54.379ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 418.513852ms iteration = 5 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
33022025-03-07T22:35:54.381ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
33032025-03-07T22:35:54.381ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
33042025-03-07T22:35:54.381ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-07 22:34:17.797135 UTC dns_group = external generation = 2
33052025-03-07T22:35:54.381ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 420.904895ms iteration = 3 status = {"generation":2}
33062025-03-07T22:35:54.488ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33072025-03-07T22:35:54.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33082025-03-07T22:35:54.490ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
33092025-03-07T22:35:54.490ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33102025-03-07T22:35:54.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33112025-03-07T22:35:54.500ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
33122025-03-07T22:35:54.501ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "3563e9ef-185b-441b-8202-86f18f58f1bd", "content-length": "691", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} })
33132025-03-07T22:35:54.501ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
33142025-03-07T22:35:54.501ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
33152025-03-07T22:35:54.501ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
33162025-03-07T22:35:54.503ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 542.350012ms iteration = 5 status = {}
33172025-03-07T22:35:54.503ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
33182025-03-07T22:35:54.503ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d8a0419-5377-4384-bbae-047409f886bf", "content-length": "134", "date": "Fri, 07 Mar 2025 22:35:53 GMT"} })
33192025-03-07T22:35:54.503ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
33202025-03-07T22:35:54.503ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
33212025-03-07T22:35:54.504ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
33222025-03-07T22:35:54.504ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
33232025-03-07T22:35:54.504ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33242025-03-07T22:35:54.504ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33252025-03-07T22:35:54.504ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33262025-03-07T22:35:54.504ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
33272025-03-07T22:35:54.504ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33282025-03-07T22:35:54.504ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
33292025-03-07T22:35:54.504ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
33302025-03-07T22:35:54.504ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33312025-03-07T22:35:54.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33322025-03-07T22:35:54.507ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
33332025-03-07T22:35:54.507ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "8eb8713f-3220-40e5-a44c-5364f8553619", "content-length": "2", "date": "Fri, 07 Mar 2025 22:35:53 GMT"} })
33342025-03-07T22:35:54.507ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
33352025-03-07T22:35:54.507ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fd4afbf-7ee5-4642-8f15-7914e282cb5b", "content-length": "52", "date": "Fri, 07 Mar 2025 22:35:53 GMT"} })
33362025-03-07T22:35:54.507ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
33372025-03-07T22:35:54.507ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
33382025-03-07T22:35:54.620ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "49db065c-83c8-4e75-8034-395f003f9937", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} })
33392025-03-07T22:35:54.620ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
33402025-03-07T22:35:54.620ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
33412025-03-07T22:35:54.621ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea3824b0-408c-4847-85a7-02d1756b58e5", "content-length": "682", "date": "Fri, 07 Mar 2025 22:35:54 GMT"} })
33422025-03-07T22:35:54.621ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33432025-03-07T22:35:54.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33442025-03-07T22:35:54.625ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33452025-03-07T22:35:54.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33462025-03-07T22:35:54.629ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33472025-03-07T22:35:54.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33482025-03-07T22:35:54.631ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33492025-03-07T22:35:54.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33502025-03-07T22:35:54.671ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 287 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 47d0e880-d9c7-4bd1-b212-3b9bb1ae8956 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
33512025-03-07T22:35:54.730ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 769.802968ms iteration = 5 status = {}
33522025-03-07T22:36:03.961ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 14 reason = Timeout
33532025-03-07T22:36:03.961ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33542025-03-07T22:36:03.961ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 14 reason = Timeout
33552025-03-07T22:36:03.961ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33562025-03-07T22:36:03.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33572025-03-07T22:36:03.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33582025-03-07T22:36:03.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33592025-03-07T22:36:03.964ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33602025-03-07T22:36:03.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33612025-03-07T22:36:03.966ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
33622025-03-07T22:36:03.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33632025-03-07T22:36:03.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33642025-03-07T22:36:03.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33652025-03-07T22:36:03.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33662025-03-07T22:36:03.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33672025-03-07T22:36:03.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
33682025-03-07T22:36:03.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33692025-03-07T22:36:03.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33702025-03-07T22:36:03.995ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.432141ms iteration = 14 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
33712025-03-07T22:36:04.084ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
33722025-03-07T22:36:04.084ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 122.506396ms iteration = 14 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
33732025-03-07T22:36:04.671ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 315 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 806608ca-11eb-4dcf-bb43-09ccb002c47e response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
33742025-03-07T22:36:13.960ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 15 reason = Timeout
33752025-03-07T22:36:13.960ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33762025-03-07T22:36:13.960ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 15 reason = Timeout
33772025-03-07T22:36:13.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33782025-03-07T22:36:13.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33792025-03-07T22:36:13.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33802025-03-07T22:36:13.963ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33812025-03-07T22:36:13.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33822025-03-07T22:36:13.999ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33832025-03-07T22:36:14.000ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 39.939124ms iteration = 15 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
33842025-03-07T22:36:14.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
33852025-03-07T22:36:14.002ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33862025-03-07T22:36:14.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33872025-03-07T22:36:14.004ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33882025-03-07T22:36:14.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33892025-03-07T22:36:14.006ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33902025-03-07T22:36:14.008ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
33912025-03-07T22:36:14.008ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33922025-03-07T22:36:14.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33932025-03-07T22:36:14.020ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
33942025-03-07T22:36:14.020ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 59.35472ms iteration = 15 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
33952025-03-07T22:36:14.671ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 293 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 57493ab6-be57-417a-816a-a88ce7dc0328 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
33962025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 16 reason = Timeout
33972025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 16 reason = Timeout
33982025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33992025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34002025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 6 reason = Timeout
34012025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 6 reason = Timeout
34022025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34032025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34042025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 6 reason = Timeout
34052025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
34062025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
34072025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 6 reason = Timeout
34082025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
34092025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 6 reason = Timeout
34102025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
34112025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 6 reason = Timeout
34122025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 6 reason = Timeout
34132025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
34142025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34152025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 6 reason = Timeout
34162025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34172025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 6 reason = Timeout
34182025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 7 reason = Timeout
34192025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
34202025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 6 reason = Timeout
34212025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 6 reason = Timeout
34222025-03-07T22:36:23.961ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
34232025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
34242025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34252025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34262025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34272025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
34282025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34292025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
34302025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
34312025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34322025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34332025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34342025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
34352025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
34362025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
34372025-03-07T22:36:23.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34382025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 6 reason = Timeout
34392025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 6 reason = Timeout
34402025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
34412025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
34422025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 6 reason = Timeout
34432025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
34442025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 6 reason = Timeout
34452025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
34462025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 6 reason = Timeout
34472025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
34482025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
34492025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
34502025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
34512025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
34522025-03-07T22:36:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
34532025-03-07T22:36:23.963ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34542025-03-07T22:36:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34552025-03-07T22:36:23.964ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34562025-03-07T22:36:23.965ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34572025-03-07T22:36:23.965ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.913857ms iteration = 6 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
34582025-03-07T22:36:23.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34592025-03-07T22:36:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
34602025-03-07T22:36:23.966ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34612025-03-07T22:36:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34622025-03-07T22:36:23.967ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 942, tv_nsec: 323987552 } })
34632025-03-07T22:36:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
34642025-03-07T22:36:23.967ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
34652025-03-07T22:36:23.967ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
34662025-03-07T22:36:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
34672025-03-07T22:36:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
34682025-03-07T22:36:23.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34692025-03-07T22:36:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34702025-03-07T22:36:23.970ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34712025-03-07T22:36:23.971ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 942, tv_nsec: 328020711 } })
34722025-03-07T22:36:23.971ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34732025-03-07T22:36:23.971ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
34742025-03-07T22:36:23.971ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 9.39668ms iteration = 6 status = {"errors":[],"garbage_collect_requested":[]}
34752025-03-07T22:36:23.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
34762025-03-07T22:36:23.972ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
34772025-03-07T22:36:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
34782025-03-07T22:36:23.973ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
34792025-03-07T22:36:23.973ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34802025-03-07T22:36:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34812025-03-07T22:36:23.976ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 15.357178ms iteration = 16 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
34822025-03-07T22:36:23.978ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
34832025-03-07T22:36:23.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
34842025-03-07T22:36:23.979ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34852025-03-07T22:36:23.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34862025-03-07T22:36:23.980ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
34872025-03-07T22:36:23.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
34882025-03-07T22:36:23.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34892025-03-07T22:36:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
34902025-03-07T22:36:23.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34912025-03-07T22:36:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34922025-03-07T22:36:24.001ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
34932025-03-07T22:36:24.001ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 40.487089ms iteration = 16 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
34942025-03-07T22:36:24.009ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34952025-03-07T22:36:24.012ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
34962025-03-07T22:36:24.012ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34972025-03-07T22:36:24.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34982025-03-07T22:36:24.018ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34992025-03-07T22:36:24.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35002025-03-07T22:36:24.019ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
35012025-03-07T22:36:24.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
35022025-03-07T22:36:24.022ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
35032025-03-07T22:36:24.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
35042025-03-07T22:36:24.023ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
35052025-03-07T22:36:24.023ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
35062025-03-07T22:36:24.023ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
35072025-03-07T22:36:24.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
35082025-03-07T22:36:24.024ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35092025-03-07T22:36:24.025ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
35102025-03-07T22:36:24.025ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
35112025-03-07T22:36:24.025ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35122025-03-07T22:36:24.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
35132025-03-07T22:36:24.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35142025-03-07T22:36:24.026ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
35152025-03-07T22:36:24.026ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
35162025-03-07T22:36:24.026ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
35172025-03-07T22:36:24.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
35182025-03-07T22:36:24.027ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
35192025-03-07T22:36:24.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
35202025-03-07T22:36:24.027ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35212025-03-07T22:36:24.028ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 66.925892ms iteration = 6 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
35222025-03-07T22:36:24.028ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 66.6484ms iteration = 6 status = {"errors":[],"finish_invoked_ok":[]}
35232025-03-07T22:36:24.029ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
35242025-03-07T22:36:24.029ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
35252025-03-07T22:36:24.029ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35262025-03-07T22:36:24.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
35272025-03-07T22:36:24.029ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 942, tv_nsec: 386502992 } })
35282025-03-07T22:36:24.029ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
35292025-03-07T22:36:24.031ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35302025-03-07T22:36:24.031ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 68.846011ms iteration = 6 status = {"errors":[],"requests_created_ok":[]}
35312025-03-07T22:36:24.031ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
35322025-03-07T22:36:24.031ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35332025-03-07T22:36:24.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35342025-03-07T22:36:24.032ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
35352025-03-07T22:36:24.032ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
35362025-03-07T22:36:24.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
35372025-03-07T22:36:24.034ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
35382025-03-07T22:36:24.034ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 73.402675ms iteration = 6 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
35392025-03-07T22:36:24.038ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
35402025-03-07T22:36:24.038ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
35412025-03-07T22:36:24.039ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "657192e7-5591-4cdd-986f-61bbf0b93add", "content-length": "26", "date": "Fri, 07 Mar 2025 22:36:24 GMT"} })
35422025-03-07T22:36:24.039ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
35432025-03-07T22:36:24.040ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
35442025-03-07T22:36:24.059ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 96.7654ms iteration = 6 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
35452025-03-07T22:36:24.067ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
35462025-03-07T22:36:24.067ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
35472025-03-07T22:36:24.067ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 106.339542ms iteration = 6 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
35482025-03-07T22:36:24.069ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
35492025-03-07T22:36:24.069ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 108.312481ms iteration = 6 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
35502025-03-07T22:36:24.076ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 113.973945ms iteration = 6 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
35512025-03-07T22:36:24.078ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
35522025-03-07T22:36:24.078ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
35532025-03-07T22:36:24.079ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa2af24f-c1cd-415c-bc0a-e2edf98cf6ba", "content-length": "26", "date": "Fri, 07 Mar 2025 22:36:24 GMT"} })
35542025-03-07T22:36:24.079ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
35552025-03-07T22:36:24.080ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
35562025-03-07T22:36:24.083ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 121.962621ms iteration = 7 status = {"physical_disks_added":0}
35572025-03-07T22:36:24.113ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35582025-03-07T22:36:24.114ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35592025-03-07T22:36:24.125ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35602025-03-07T22:36:24.125ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35612025-03-07T22:36:24.144ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
35622025-03-07T22:36:24.144ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
35632025-03-07T22:36:24.144ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "67ede3fe-018a-46c1-a44e-d6cbf24caa39", "content-length": "26", "date": "Fri, 07 Mar 2025 22:36:24 GMT"} })
35642025-03-07T22:36:24.144ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
35652025-03-07T22:36:24.145ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
35662025-03-07T22:36:24.150ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
35672025-03-07T22:36:24.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35682025-03-07T22:36:24.161ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35692025-03-07T22:36:24.163ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
35702025-03-07T22:36:24.163ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35712025-03-07T22:36:24.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35722025-03-07T22:36:24.177ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
35732025-03-07T22:36:24.178ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2004b5b-bbf5-4126-acc8-7a72de2ddd42", "content-length": "1", "date": "Fri, 07 Mar 2025 22:36:23 GMT"} })
35742025-03-07T22:36:24.179ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
35752025-03-07T22:36:24.179ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
35762025-03-07T22:36:24.238ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
35772025-03-07T22:36:24.238ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
35782025-03-07T22:36:24.239ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "10aaebb4-a7b3-4b22-87fc-4f30ad3c37e7", "content-length": "372", "date": "Fri, 07 Mar 2025 22:36:24 GMT"} })
35792025-03-07T22:36:24.239ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35802025-03-07T22:36:24.239ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35812025-03-07T22:36:24.241ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35822025-03-07T22:36:24.241ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
35832025-03-07T22:36:24.241ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35842025-03-07T22:36:24.241ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 280.748478ms iteration = 6 status = null
35852025-03-07T22:36:24.242ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
35862025-03-07T22:36:24.242ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35872025-03-07T22:36:24.242ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35882025-03-07T22:36:24.243ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35892025-03-07T22:36:24.243ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 282.692836ms iteration = 6 status = {}
35902025-03-07T22:36:24.245ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
35912025-03-07T22:36:24.245ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
35922025-03-07T22:36:24.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35932025-03-07T22:36:24.247ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 286.461153ms iteration = 6 status = 0
35942025-03-07T22:36:24.254ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
35952025-03-07T22:36:24.254ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
35962025-03-07T22:36:24.254ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "6ca65ed0-02d9-48a1-8a25-37c2b7b262e7", "date": "Fri, 07 Mar 2025 22:36:24 GMT"} })
35972025-03-07T22:36:24.254ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35982025-03-07T22:36:24.254ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35992025-03-07T22:36:24.259ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
36002025-03-07T22:36:24.259ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36012025-03-07T22:36:24.262ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
36022025-03-07T22:36:24.262ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36032025-03-07T22:36:24.265ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
36042025-03-07T22:36:24.265ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
36052025-03-07T22:36:24.265ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "c413b330-1321-46c3-a13f-51407e3e5012", "date": "Fri, 07 Mar 2025 22:36:24 GMT"} })
36062025-03-07T22:36:24.266ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 305.755308ms iteration = 6 status = {}
36072025-03-07T22:36:24.304ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36082025-03-07T22:36:24.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36092025-03-07T22:36:24.307ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
36102025-03-07T22:36:24.307ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36112025-03-07T22:36:24.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36122025-03-07T22:36:24.318ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
36132025-03-07T22:36:24.318ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9136b94-0bc7-475c-8a34-6214c67a156f", "content-length": "134", "date": "Fri, 07 Mar 2025 22:36:23 GMT"} })
36142025-03-07T22:36:24.318ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
36152025-03-07T22:36:24.318ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
36162025-03-07T22:36:24.318ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
36172025-03-07T22:36:24.318ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
36182025-03-07T22:36:24.319ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36192025-03-07T22:36:24.319ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36202025-03-07T22:36:24.319ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36212025-03-07T22:36:24.319ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
36222025-03-07T22:36:24.319ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36232025-03-07T22:36:24.319ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
36242025-03-07T22:36:24.319ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
36252025-03-07T22:36:24.319ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36262025-03-07T22:36:24.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36272025-03-07T22:36:24.321ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
36282025-03-07T22:36:24.321ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "7572dc6d-cc9b-40c0-9e3d-e138c93dc0cf", "content-length": "2", "date": "Fri, 07 Mar 2025 22:36:23 GMT"} })
36292025-03-07T22:36:24.321ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
36302025-03-07T22:36:24.321ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "339ea358-1709-4e2c-a3ab-04fc7aa321df", "content-length": "52", "date": "Fri, 07 Mar 2025 22:36:23 GMT"} })
36312025-03-07T22:36:24.321ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
36322025-03-07T22:36:24.321ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
36332025-03-07T22:36:24.337ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
36342025-03-07T22:36:24.416ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "2a7a0ce9-d5e3-4fb2-8dbc-dce53f66c7e9", "date": "Fri, 07 Mar 2025 22:36:24 GMT"} })
36352025-03-07T22:36:24.416ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
36362025-03-07T22:36:24.416ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
36372025-03-07T22:36:24.417ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "90388766-b7f7-4556-9e93-27fc8f9921ac", "content-length": "682", "date": "Fri, 07 Mar 2025 22:36:24 GMT"} })
36382025-03-07T22:36:24.417ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36392025-03-07T22:36:24.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36402025-03-07T22:36:24.417ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0871c23a-972c-4876-a3e8-b67eea34baed", "content-length": "691", "date": "Fri, 07 Mar 2025 22:36:24 GMT"} })
36412025-03-07T22:36:24.417ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
36422025-03-07T22:36:24.418ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
36432025-03-07T22:36:24.418ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
36442025-03-07T22:36:24.419ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 458.007631ms iteration = 6 status = {}
36452025-03-07T22:36:24.439ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36462025-03-07T22:36:24.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36472025-03-07T22:36:24.448ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36482025-03-07T22:36:24.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36492025-03-07T22:36:24.450ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36502025-03-07T22:36:24.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36512025-03-07T22:36:24.457ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 496.612661ms iteration = 6 status = {}
36522025-03-07T22:36:24.671ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 327 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 04cbf8a5-971e-48bd-9106-aa3a1c5d061e response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
36532025-03-07T22:36:33.961ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 17 reason = Timeout
36542025-03-07T22:36:33.961ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 17 reason = Timeout
36552025-03-07T22:36:33.961ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36562025-03-07T22:36:33.961ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36572025-03-07T22:36:33.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36582025-03-07T22:36:33.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36592025-03-07T22:36:33.964ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36602025-03-07T22:36:33.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36612025-03-07T22:36:33.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36622025-03-07T22:36:33.966ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.754916ms iteration = 17 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
36632025-03-07T22:36:33.966ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
36642025-03-07T22:36:33.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36652025-03-07T22:36:33.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36662025-03-07T22:36:33.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36672025-03-07T22:36:33.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36682025-03-07T22:36:33.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36692025-03-07T22:36:33.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
36702025-03-07T22:36:33.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36712025-03-07T22:36:33.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36722025-03-07T22:36:33.986ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
36732025-03-07T22:36:33.986ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.661247ms iteration = 17 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
36742025-03-07T22:36:34.671ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 284 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 123e8277-69bc-40a7-9bd8-d8f7a8926295 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
36752025-03-07T22:36:43.961ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 18 reason = Timeout
36762025-03-07T22:36:43.961ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 18 reason = Timeout
36772025-03-07T22:36:43.961ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36782025-03-07T22:36:43.961ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36792025-03-07T22:36:43.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36802025-03-07T22:36:43.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36812025-03-07T22:36:43.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36822025-03-07T22:36:43.963ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36832025-03-07T22:36:43.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36842025-03-07T22:36:43.965ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.1428ms iteration = 18 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
36852025-03-07T22:36:43.966ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
36862025-03-07T22:36:43.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36872025-03-07T22:36:43.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36882025-03-07T22:36:43.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36892025-03-07T22:36:43.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36902025-03-07T22:36:43.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36912025-03-07T22:36:43.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
36922025-03-07T22:36:43.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36932025-03-07T22:36:43.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36942025-03-07T22:36:43.982ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
36952025-03-07T22:36:43.982ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.050192ms iteration = 18 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
36962025-03-07T22:36:44.672ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 253 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 4ae14434-ef75-4aac-869e-190844d14041 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
36972025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 19 reason = Timeout
36982025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36992025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 7 reason = Timeout
37002025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 4 reason = Timeout
37012025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
37022025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 7 reason = Timeout
37032025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
37042025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 5 reason = Timeout
37052025-03-07T22:36:53.961ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
37062025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
37072025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 4 reason = Timeout
37082025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.041\u{b5}s iteration = 5 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
37092025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 7 reason = Timeout
37102025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 4 reason = Timeout
37112025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
37122025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
37132025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
37142025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 6 reason = Timeout
37152025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 4 reason = Timeout
37162025-03-07T22:36:53.961ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
37172025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 7 reason = Timeout
37182025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
37192025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 4 reason = Timeout
37202025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
37212025-03-07T22:36:53.961ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
37222025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 4 reason = Timeout
37232025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
37242025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 7 reason = Timeout
37252025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 7 reason = Timeout
37262025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
37272025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 4 reason = Timeout
37282025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
37292025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
37302025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 3 reason = Timeout
37312025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 7 reason = Timeout
37322025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 6 reason = Timeout
37332025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
37342025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 4 reason = Timeout
37352025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37362025-03-07T22:36:53.962ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:26:53.961974478 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
37372025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
37382025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 7 reason = Timeout
37392025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
37402025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 19 reason = Timeout
37412025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
37422025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 7 reason = Timeout
37432025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 7 reason = Timeout
37442025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37452025-03-07T22:36:53.962ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
37462025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37472025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 8 reason = Timeout
37482025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37492025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 7 reason = Timeout
37502025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37512025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 7 reason = Timeout
37522025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
37532025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 7 reason = Timeout
37542025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
37552025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 7 reason = Timeout
37562025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
37572025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 4 reason = Timeout
37582025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
37592025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 7 reason = Timeout
37602025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
37612025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 7 reason = Timeout
37622025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
37632025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 4 reason = Timeout
37642025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
37652025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
37662025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
37672025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
37682025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37692025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
37702025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
37712025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
37722025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
37732025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37742025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
37752025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
37762025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
37772025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37782025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
37792025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
37802025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
37812025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
37822025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37832025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37842025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37852025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
37862025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37872025-03-07T22:36:53.962ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 972, tv_nsec: 319185946 } })
37882025-03-07T22:36:53.962ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 972, tv_nsec: 319194526 } })
37892025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
37902025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37912025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37922025-03-07T22:36:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
37932025-03-07T22:36:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
37942025-03-07T22:36:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
37952025-03-07T22:36:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
37962025-03-07T22:36:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
37972025-03-07T22:36:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37982025-03-07T22:36:53.963ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37992025-03-07T22:36:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
38002025-03-07T22:36:53.963ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
38012025-03-07T22:36:53.963ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
38022025-03-07T22:36:53.963ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.502945ms iteration = 4 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
38032025-03-07T22:36:53.963ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
38042025-03-07T22:36:53.963ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.598445ms iteration = 4 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
38052025-03-07T22:36:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
38062025-03-07T22:36:53.965ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38072025-03-07T22:36:53.965ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
38082025-03-07T22:36:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
38092025-03-07T22:36:53.965ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38102025-03-07T22:36:53.965ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.670875ms iteration = 7 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
38112025-03-07T22:36:53.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38122025-03-07T22:36:53.967ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
38132025-03-07T22:36:53.967ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
38142025-03-07T22:36:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
38152025-03-07T22:36:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
38162025-03-07T22:36:53.967ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
38172025-03-07T22:36:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
38182025-03-07T22:36:53.967ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
38192025-03-07T22:36:53.968ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
38202025-03-07T22:36:53.968ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
38212025-03-07T22:36:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
38222025-03-07T22:36:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
38232025-03-07T22:36:53.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38242025-03-07T22:36:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38252025-03-07T22:36:53.969ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38262025-03-07T22:36:53.969ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
38272025-03-07T22:36:53.969ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38282025-03-07T22:36:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
38292025-03-07T22:36:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
38302025-03-07T22:36:53.970ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
38312025-03-07T22:36:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
38322025-03-07T22:36:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
38332025-03-07T22:36:53.971ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38342025-03-07T22:36:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38352025-03-07T22:36:53.972ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 972, tv_nsec: 328426935 } })
38362025-03-07T22:36:53.972ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
38372025-03-07T22:36:53.972ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38382025-03-07T22:36:53.973ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38392025-03-07T22:36:53.974ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:26:53.961974478 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
38402025-03-07T22:36:53.974ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 12.578671ms iteration = 4 status = {"expiration":"2025-03-07T22:26:53.961974478Z","pruned":[]}
38412025-03-07T22:36:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
38422025-03-07T22:36:53.974ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38432025-03-07T22:36:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38442025-03-07T22:36:53.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
38452025-03-07T22:36:53.975ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
38462025-03-07T22:36:53.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
38472025-03-07T22:36:53.976ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38482025-03-07T22:36:53.977ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 972, tv_nsec: 333411923 } })
38492025-03-07T22:36:53.977ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38502025-03-07T22:36:53.977ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
38512025-03-07T22:36:53.978ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
38522025-03-07T22:36:53.978ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
38532025-03-07T22:36:53.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
38542025-03-07T22:36:53.979ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
38552025-03-07T22:36:53.979ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38562025-03-07T22:36:53.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38572025-03-07T22:36:53.983ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 21.199304ms iteration = 7 status = {"errors":[],"garbage_collect_requested":[]}
38582025-03-07T22:36:53.985ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38592025-03-07T22:36:53.986ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38602025-03-07T22:36:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
38612025-03-07T22:36:53.988ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
38622025-03-07T22:36:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
38632025-03-07T22:36:53.988ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38642025-03-07T22:36:53.990ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
38652025-03-07T22:36:53.990ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
38662025-03-07T22:36:53.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
38672025-03-07T22:36:53.991ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38682025-03-07T22:36:53.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
38692025-03-07T22:36:53.992ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
38702025-03-07T22:36:53.992ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
38712025-03-07T22:36:53.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
38722025-03-07T22:36:53.994ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
38732025-03-07T22:36:53.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
38742025-03-07T22:36:53.995ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38752025-03-07T22:36:53.996ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38762025-03-07T22:36:53.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38772025-03-07T22:36:53.999ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
38782025-03-07T22:36:53.999ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 37.946935ms iteration = 7 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
38792025-03-07T22:36:53.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
38802025-03-07T22:36:54.001ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38812025-03-07T22:36:54.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38822025-03-07T22:36:54.015ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
38832025-03-07T22:36:54.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
38842025-03-07T22:36:54.015ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
38852025-03-07T22:36:54.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
38862025-03-07T22:36:54.019ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38872025-03-07T22:36:54.020ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
38882025-03-07T22:36:54.020ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38892025-03-07T22:36:54.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38902025-03-07T22:36:54.023ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38912025-03-07T22:36:54.025ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 63.206077ms iteration = 7 status = {"errors":[],"requests_created_ok":[]}
38922025-03-07T22:36:54.025ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
38932025-03-07T22:36:54.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
38942025-03-07T22:36:54.026ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
38952025-03-07T22:36:54.026ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
38962025-03-07T22:36:54.026ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
38972025-03-07T22:36:54.026ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
38982025-03-07T22:36:54.026ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
38992025-03-07T22:36:54.026ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
39002025-03-07T22:36:54.026ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
39012025-03-07T22:36:54.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
39022025-03-07T22:36:54.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
39032025-03-07T22:36:54.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
39042025-03-07T22:36:54.027ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39052025-03-07T22:36:54.029ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
39062025-03-07T22:36:54.029ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
39072025-03-07T22:36:54.029ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
39082025-03-07T22:36:54.029ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
39092025-03-07T22:36:54.029ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39102025-03-07T22:36:54.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39112025-03-07T22:36:54.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
39122025-03-07T22:36:54.030ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
39132025-03-07T22:36:54.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
39142025-03-07T22:36:54.033ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39152025-03-07T22:36:54.035ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
39162025-03-07T22:36:54.035ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
39172025-03-07T22:36:54.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
39182025-03-07T22:36:54.037ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 75.198192ms iteration = 7 status = {"errors":[],"finish_invoked_ok":[]}
39192025-03-07T22:36:54.037ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 75.392744ms iteration = 7 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
39202025-03-07T22:36:54.038ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
39212025-03-07T22:36:54.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
39222025-03-07T22:36:54.039ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 972, tv_nsec: 395611290 } })
39232025-03-07T22:36:54.039ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
39242025-03-07T22:36:54.039ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
39252025-03-07T22:36:54.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
39262025-03-07T22:36:54.041ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
39272025-03-07T22:36:54.041ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
39282025-03-07T22:36:54.041ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 79.04114ms iteration = 4 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
39292025-03-07T22:36:54.041ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39302025-03-07T22:36:54.041ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 79.681175ms iteration = 3 status = {"nsuccess":5}
39312025-03-07T22:36:54.042ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 80.041469ms iteration = 7 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
39322025-03-07T22:36:54.043ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
39332025-03-07T22:36:54.043ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
39342025-03-07T22:36:54.043ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
39352025-03-07T22:36:54.043ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
39362025-03-07T22:36:54.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
39372025-03-07T22:36:54.043ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
39382025-03-07T22:36:54.043ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
39392025-03-07T22:36:54.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
39402025-03-07T22:36:54.046ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39412025-03-07T22:36:54.046ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
39422025-03-07T22:36:54.046ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
39432025-03-07T22:36:54.046ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 84.616173ms iteration = 4 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
39442025-03-07T22:36:54.049ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
39452025-03-07T22:36:54.049ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
39462025-03-07T22:36:54.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
39472025-03-07T22:36:54.063ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
39482025-03-07T22:36:54.065ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
39492025-03-07T22:36:54.080ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 117.973463ms iteration = 19 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
39502025-03-07T22:36:54.082ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
39512025-03-07T22:36:54.082ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
39522025-03-07T22:36:54.082ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 120.324656ms iteration = 7 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
39532025-03-07T22:36:54.082ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39542025-03-07T22:36:54.085ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
39552025-03-07T22:36:54.085ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
39562025-03-07T22:36:54.086ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
39572025-03-07T22:36:54.089ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
39582025-03-07T22:36:54.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
39592025-03-07T22:36:54.092ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 130.029689ms iteration = 8 status = {"physical_disks_added":0}
39602025-03-07T22:36:54.092ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 130.513144ms iteration = 7 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
39612025-03-07T22:36:54.092ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
39622025-03-07T22:36:54.092ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
39632025-03-07T22:36:54.092ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39642025-03-07T22:36:54.092ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb4d0b16-4ff2-40a8-bcc7-6ee6fdb48071", "content-length": "26", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} })
39652025-03-07T22:36:54.092ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
39662025-03-07T22:36:54.095ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
39672025-03-07T22:36:54.095ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39682025-03-07T22:36:54.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39692025-03-07T22:36:54.095ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
39702025-03-07T22:36:54.096ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 134.018288ms iteration = 4 status = {"errors":[],"found_port_ok":[]}
39712025-03-07T22:36:54.096ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39722025-03-07T22:36:54.101ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
39732025-03-07T22:36:54.101ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
39742025-03-07T22:36:54.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
39752025-03-07T22:36:54.103ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
39762025-03-07T22:36:54.103ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 141.643551ms iteration = 7 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
39772025-03-07T22:36:54.124ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
39782025-03-07T22:36:54.124ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 163.065946ms iteration = 19 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
39792025-03-07T22:36:54.135ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
39802025-03-07T22:36:54.135ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
39812025-03-07T22:36:54.136ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "53a2eed9-5987-4753-8bbc-ecc089e8f439", "content-length": "26", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} })
39822025-03-07T22:36:54.136ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
39832025-03-07T22:36:54.137ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
39842025-03-07T22:36:54.146ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39852025-03-07T22:36:54.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39862025-03-07T22:36:54.149ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
39872025-03-07T22:36:54.149ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
39882025-03-07T22:36:54.156ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39892025-03-07T22:36:54.156ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
39902025-03-07T22:36:54.156ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
39912025-03-07T22:36:54.156ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
39922025-03-07T22:36:54.156ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39932025-03-07T22:36:54.156ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
39942025-03-07T22:36:54.177ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39952025-03-07T22:36:54.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
39962025-03-07T22:36:54.204ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39972025-03-07T22:36:54.206ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
39982025-03-07T22:36:54.206ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
39992025-03-07T22:36:54.206ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
40002025-03-07T22:36:54.224ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
40012025-03-07T22:36:54.224ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
40022025-03-07T22:36:54.224ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
40032025-03-07T22:36:54.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
40042025-03-07T22:36:54.226ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
40052025-03-07T22:36:54.226ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
40062025-03-07T22:36:54.226ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "98ae54cc-6469-40cf-8b55-8d5d940d2e4f", "content-length": "26", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} })
40072025-03-07T22:36:54.226ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
40082025-03-07T22:36:54.228ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
40092025-03-07T22:36:54.232ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
40102025-03-07T22:36:54.232ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 270.214707ms iteration = 4 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
40112025-03-07T22:36:54.238ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
40122025-03-07T22:36:54.239ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "4bf3cdc6-f7cb-4c7c-b671-a5188b37ad46", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
40132025-03-07T22:36:54.239ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
40142025-03-07T22:36:54.240ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "bbb3ba01-1d3b-4eae-a945-1c4cc87b6ee2", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
40152025-03-07T22:36:54.240ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
40162025-03-07T22:36:54.242ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
40172025-03-07T22:36:54.242ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 280.291353ms iteration = 6 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
40182025-03-07T22:36:54.309ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 347.195157ms iteration = 7 status = {}
40192025-03-07T22:36:54.311ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
40202025-03-07T22:36:54.314ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
40212025-03-07T22:36:54.314ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
40222025-03-07T22:36:54.315ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "86f9e2d3-0536-480c-b21c-dba48758824c", "content-length": "372", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} })
40232025-03-07T22:36:54.315ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40242025-03-07T22:36:54.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40252025-03-07T22:36:54.316ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40262025-03-07T22:36:54.317ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40272025-03-07T22:36:54.318ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
40282025-03-07T22:36:54.318ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
40292025-03-07T22:36:54.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
40302025-03-07T22:36:54.318ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40312025-03-07T22:36:54.318ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
40322025-03-07T22:36:54.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
40332025-03-07T22:36:54.321ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
40342025-03-07T22:36:54.321ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
40352025-03-07T22:36:54.321ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
40362025-03-07T22:36:54.321ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
40372025-03-07T22:36:54.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
40382025-03-07T22:36:54.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
40392025-03-07T22:36:54.324ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
40402025-03-07T22:36:54.324ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
40412025-03-07T22:36:54.324ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-07 22:34:17.797135 UTC dns_group = external generation = 2
40422025-03-07T22:36:54.324ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 362.809717ms iteration = 4 status = {"generation":2}
40432025-03-07T22:36:54.325ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
40442025-03-07T22:36:54.325ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
40452025-03-07T22:36:54.325ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-07 22:34:17.208516 UTC dns_group = internal generation = 1
40462025-03-07T22:36:54.325ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 363.349732ms iteration = 4 status = {"generation":1}
40472025-03-07T22:36:54.359ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40482025-03-07T22:36:54.361ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
40492025-03-07T22:36:54.361ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
40502025-03-07T22:36:54.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
40512025-03-07T22:36:54.364ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40522025-03-07T22:36:54.367ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
40532025-03-07T22:36:54.367ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
40542025-03-07T22:36:54.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
40552025-03-07T22:36:54.369ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 407.707798ms iteration = 7 status = 0
40562025-03-07T22:36:54.380ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
40572025-03-07T22:36:54.381ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "cae9971b-1356-4ee8-956c-f081b5a75368", "content-length": "1", "date": "Fri, 07 Mar 2025 22:36:53 GMT"} })
40582025-03-07T22:36:54.382ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
40592025-03-07T22:36:54.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
40602025-03-07T22:36:54.384ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 422.988504ms iteration = 7 status = null
40612025-03-07T22:36:54.405ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
40622025-03-07T22:36:54.405ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "626ecbf4-0a37-4a6b-ab88-afa6a32bc405", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40632025-03-07T22:36:54.405ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40642025-03-07T22:36:54.406ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "b90fe239-9e7f-4153-a1e7-ded263805d4b", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40652025-03-07T22:36:54.406ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40662025-03-07T22:36:54.408ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "7638c6df-2fe8-49f1-bf1c-b9bedf21d871", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40672025-03-07T22:36:54.408ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40682025-03-07T22:36:54.410ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40692025-03-07T22:36:54.410ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 449.008515ms iteration = 6 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
40702025-03-07T22:36:54.472ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) }
40712025-03-07T22:36:54.472ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 510.421435ms iteration = 4 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
40722025-03-07T22:36:54.489ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
40732025-03-07T22:36:54.489ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
40742025-03-07T22:36:54.490ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "0e1b8926-0618-4d68-aaee-a79ebccac253", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} })
40752025-03-07T22:36:54.490ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40762025-03-07T22:36:54.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40772025-03-07T22:36:54.493ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40782025-03-07T22:36:54.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40792025-03-07T22:36:54.494ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40802025-03-07T22:36:54.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40812025-03-07T22:36:54.496ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40822025-03-07T22:36:54.496ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40832025-03-07T22:36:54.523ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
40842025-03-07T22:36:54.524ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "726b25e5-fa56-449f-a456-4300560094bc", "content-length": "691", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} })
40852025-03-07T22:36:54.524ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
40862025-03-07T22:36:54.524ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
40872025-03-07T22:36:54.524ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
40882025-03-07T22:36:54.525ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 563.770388ms iteration = 7 status = {}
40892025-03-07T22:36:54.526ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
40902025-03-07T22:36:54.526ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40912025-03-07T22:36:54.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40922025-03-07T22:36:54.528ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
40932025-03-07T22:36:54.528ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
40942025-03-07T22:36:54.528ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "33713656-df6d-4171-bf49-178aad1d815f", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} })
40952025-03-07T22:36:54.530ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 568.075509ms iteration = 7 status = {}
40962025-03-07T22:36:54.551ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
40972025-03-07T22:36:54.551ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "8321a182-4867-4591-9377-7b6bc7682c59", "content-length": "134", "date": "Fri, 07 Mar 2025 22:36:53 GMT"} })
40982025-03-07T22:36:54.552ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
40992025-03-07T22:36:54.552ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
41002025-03-07T22:36:54.552ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
41012025-03-07T22:36:54.552ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
41022025-03-07T22:36:54.552ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
41032025-03-07T22:36:54.552ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
41042025-03-07T22:36:54.552ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
41052025-03-07T22:36:54.552ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
41062025-03-07T22:36:54.552ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
41072025-03-07T22:36:54.552ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
41082025-03-07T22:36:54.552ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
41092025-03-07T22:36:54.552ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41102025-03-07T22:36:54.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41112025-03-07T22:36:54.555ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
41122025-03-07T22:36:54.555ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "67ef989b-fcd9-4d60-b8ab-a2e0f41b1718", "content-length": "2", "date": "Fri, 07 Mar 2025 22:36:53 GMT"} })
41132025-03-07T22:36:54.555ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
41142025-03-07T22:36:54.555ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d79fd10-6f49-4db8-bf64-3823c99ff45a", "content-length": "52", "date": "Fri, 07 Mar 2025 22:36:53 GMT"} })
41152025-03-07T22:36:54.555ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
41162025-03-07T22:36:54.555ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
41172025-03-07T22:36:54.671ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 275 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 5dc37511-e603-46b9-b951-e27d9aa67d9d response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
41182025-03-07T22:36:54.673ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "6555bbc1-13ea-4240-8b6d-45e68ec3cb63", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} })
41192025-03-07T22:36:54.673ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
41202025-03-07T22:36:54.673ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
41212025-03-07T22:36:54.673ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e858f54-7641-4996-b960-130d29855a6d", "content-length": "682", "date": "Fri, 07 Mar 2025 22:36:54 GMT"} })
41222025-03-07T22:36:54.673ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41232025-03-07T22:36:54.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41242025-03-07T22:36:54.677ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41252025-03-07T22:36:54.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41262025-03-07T22:36:54.681ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41272025-03-07T22:36:54.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41282025-03-07T22:36:54.684ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41292025-03-07T22:36:54.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41302025-03-07T22:36:54.690ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 728.626082ms iteration = 7 status = {}
41312025-03-07T22:37:03.961ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 20 reason = Timeout
41322025-03-07T22:37:03.961ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 20 reason = Timeout
41332025-03-07T22:37:03.961ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41342025-03-07T22:37:03.961ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41352025-03-07T22:37:03.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41362025-03-07T22:37:03.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41372025-03-07T22:37:03.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41382025-03-07T22:37:03.966ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
41392025-03-07T22:37:03.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41402025-03-07T22:37:03.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41412025-03-07T22:37:03.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41422025-03-07T22:37:03.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41432025-03-07T22:37:03.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41442025-03-07T22:37:03.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
41452025-03-07T22:37:03.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41462025-03-07T22:37:03.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41472025-03-07T22:37:03.984ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
41482025-03-07T22:37:03.984ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.928091ms iteration = 20 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
41492025-03-07T22:37:04.002ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41502025-03-07T22:37:04.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41512025-03-07T22:37:04.032ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 70.245125ms iteration = 20 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
41522025-03-07T22:37:04.671ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 283 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 82ccd62c-1e39-480c-b55a-383658b46116 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
41532025-03-07T22:37:13.962ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 21 reason = Timeout
41542025-03-07T22:37:13.962ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 21 reason = Timeout
41552025-03-07T22:37:13.962ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41562025-03-07T22:37:13.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41572025-03-07T22:37:13.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41582025-03-07T22:37:13.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41592025-03-07T22:37:13.994ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41602025-03-07T22:37:13.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41612025-03-07T22:37:13.997ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41622025-03-07T22:37:14.000ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
41632025-03-07T22:37:14.000ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41642025-03-07T22:37:14.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41652025-03-07T22:37:14.001ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41662025-03-07T22:37:14.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41672025-03-07T22:37:14.003ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41682025-03-07T22:37:14.005ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
41692025-03-07T22:37:14.005ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41702025-03-07T22:37:14.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41712025-03-07T22:37:14.017ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
41722025-03-07T22:37:14.017ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 55.16631ms iteration = 21 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
41732025-03-07T22:37:14.025ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 63.719472ms iteration = 21 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
41742025-03-07T22:37:14.672ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 245 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 0754f840-1fb1-4ebb-b6f3-fe1c80b6a343 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
41752025-03-07T22:37:19.662ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:49063
41762025-03-07T22:37:19.662ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = 74d6b6f1-15e4-4014-89ee-c01fbe154971 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
41772025-03-07T22:37:19.663ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = 74d6b6f1-15e4-4014-89ee-c01fbe154971 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
41782025-03-07T22:37:19.665ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2520 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = 74d6b6f1-15e4-4014-89ee-c01fbe154971 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
41792025-03-07T22:37:19.665ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = 5885f6a6-ad3e-48a2-8d82-987bb90462a5 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
41802025-03-07T22:37:19.665ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = 5885f6a6-ad3e-48a2-8d82-987bb90462a5 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
41812025-03-07T22:37:19.667ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2326 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = 5885f6a6-ad3e-48a2-8d82-987bb90462a5 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
41822025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 22 reason = Timeout
41832025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 22 reason = Timeout
41842025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41852025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41862025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 8 reason = Timeout
41872025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
41882025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
41892025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 9 reason = Timeout
41902025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 8 reason = Timeout
41912025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
41922025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
41932025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 8 reason = Timeout
41942025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
41952025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 8 reason = Timeout
41962025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
41972025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 8 reason = Timeout
41982025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 8 reason = Timeout
41992025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42002025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 8 reason = Timeout
42012025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 8 reason = Timeout
42022025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 8 reason = Timeout
42032025-03-07T22:37:23.962ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
42042025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
42052025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 8 reason = Timeout
42062025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42072025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42082025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
42092025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 8 reason = Timeout
42102025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42112025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
42122025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42132025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
42142025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
42152025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
42162025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
42172025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
42182025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
42192025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42202025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42212025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
42222025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42232025-03-07T22:37:23.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42242025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 8 reason = Timeout
42252025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 8 reason = Timeout
42262025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
42272025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 8 reason = Timeout
42282025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
42292025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
42302025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 8 reason = Timeout
42312025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
42322025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 8 reason = Timeout
42332025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
42342025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
42352025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
42362025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
42372025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
42382025-03-07T22:37:23.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
42392025-03-07T22:37:23.964ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
42402025-03-07T22:37:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
42412025-03-07T22:37:23.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42422025-03-07T22:37:23.965ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.228441ms iteration = 8 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
42432025-03-07T22:37:23.965ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42442025-03-07T22:37:23.967ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1002, tv_nsec: 323316236 } })
42452025-03-07T22:37:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
42462025-03-07T22:37:23.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42472025-03-07T22:37:23.967ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
42482025-03-07T22:37:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42492025-03-07T22:37:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
42502025-03-07T22:37:23.968ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
42512025-03-07T22:37:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
42522025-03-07T22:37:23.970ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1002, tv_nsec: 326442316 } })
42532025-03-07T22:37:23.970ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42542025-03-07T22:37:23.971ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
42552025-03-07T22:37:23.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
42562025-03-07T22:37:23.972ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42572025-03-07T22:37:23.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42582025-03-07T22:37:23.978ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42592025-03-07T22:37:23.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42602025-03-07T22:37:23.979ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1002, tv_nsec: 335557603 } })
42612025-03-07T22:37:23.980ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
42622025-03-07T22:37:23.981ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
42632025-03-07T22:37:23.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
42642025-03-07T22:37:24.008ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
42652025-03-07T22:37:24.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
42662025-03-07T22:37:24.008ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 45.609948ms iteration = 8 status = {"errors":[],"finish_invoked_ok":[]}
42672025-03-07T22:37:24.010ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42682025-03-07T22:37:24.012ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
42692025-03-07T22:37:24.012ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
42702025-03-07T22:37:24.012ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
42712025-03-07T22:37:24.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
42722025-03-07T22:37:24.012ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
42732025-03-07T22:37:24.012ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
42742025-03-07T22:37:24.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
42752025-03-07T22:37:24.014ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42762025-03-07T22:37:24.014ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42772025-03-07T22:37:24.016ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42782025-03-07T22:37:24.016ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
42792025-03-07T22:37:24.016ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
42802025-03-07T22:37:24.016ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
42812025-03-07T22:37:24.016ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42822025-03-07T22:37:24.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
42832025-03-07T22:37:24.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42842025-03-07T22:37:24.019ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
42852025-03-07T22:37:24.019ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42862025-03-07T22:37:24.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42872025-03-07T22:37:24.021ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
42882025-03-07T22:37:24.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
42892025-03-07T22:37:24.022ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42902025-03-07T22:37:24.022ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42912025-03-07T22:37:24.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42922025-03-07T22:37:24.023ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
42932025-03-07T22:37:24.023ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 60.503741ms iteration = 8 status = {"errors":[],"garbage_collect_requested":[]}
42942025-03-07T22:37:24.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
42952025-03-07T22:37:24.024ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
42962025-03-07T22:37:24.024ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42972025-03-07T22:37:24.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42982025-03-07T22:37:24.025ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42992025-03-07T22:37:24.026ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
43002025-03-07T22:37:24.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
43012025-03-07T22:37:24.027ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 65.284917ms iteration = 22 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
43022025-03-07T22:37:24.028ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
43032025-03-07T22:37:24.028ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
43042025-03-07T22:37:24.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
43052025-03-07T22:37:24.028ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 65.507099ms iteration = 8 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
43062025-03-07T22:37:24.031ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 67.850671ms iteration = 8 status = {"errors":[],"requests_created_ok":[]}
43072025-03-07T22:37:24.031ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43082025-03-07T22:37:24.031ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
43092025-03-07T22:37:24.031ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
43102025-03-07T22:37:24.031ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ed5d949-be46-4b5f-9f64-66a8de731f95", "content-length": "26", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
43112025-03-07T22:37:24.031ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
43122025-03-07T22:37:24.034ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
43132025-03-07T22:37:24.035ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
43142025-03-07T22:37:24.035ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43152025-03-07T22:37:24.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43162025-03-07T22:37:24.049ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
43172025-03-07T22:37:24.049ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
43182025-03-07T22:37:24.050ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "427f03d0-6ef0-4066-a5c6-505abdb1ceb5", "content-length": "26", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
43192025-03-07T22:37:24.050ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
43202025-03-07T22:37:24.051ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
43212025-03-07T22:37:24.051ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
43222025-03-07T22:37:24.051ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 89.783602ms iteration = 22 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
43232025-03-07T22:37:24.053ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
43242025-03-07T22:37:24.053ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
43252025-03-07T22:37:24.053ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
43262025-03-07T22:37:24.053ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
43272025-03-07T22:37:24.055ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
43282025-03-07T22:37:24.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
43292025-03-07T22:37:24.058ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
43302025-03-07T22:37:24.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
43312025-03-07T22:37:24.058ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
43322025-03-07T22:37:24.058ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
43332025-03-07T22:37:24.058ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
43342025-03-07T22:37:24.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
43352025-03-07T22:37:24.059ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa9f20e1-d697-4f1e-90e2-842cd27307fe", "content-length": "26", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
43362025-03-07T22:37:24.059ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
43372025-03-07T22:37:24.059ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 97.662769ms iteration = 8 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
43382025-03-07T22:37:24.060ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
43392025-03-07T22:37:24.060ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43402025-03-07T22:37:24.061ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43412025-03-07T22:37:24.063ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
43422025-03-07T22:37:24.063ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
43432025-03-07T22:37:24.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
43442025-03-07T22:37:24.064ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
43452025-03-07T22:37:24.064ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
43462025-03-07T22:37:24.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
43472025-03-07T22:37:24.066ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 102.739807ms iteration = 8 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
43482025-03-07T22:37:24.066ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 104.747706ms iteration = 8 status = 0
43492025-03-07T22:37:24.070ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
43502025-03-07T22:37:24.071ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 108.747774ms iteration = 8 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
43512025-03-07T22:37:24.103ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
43522025-03-07T22:37:24.103ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
43532025-03-07T22:37:24.103ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 141.408849ms iteration = 8 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
43542025-03-07T22:37:24.114ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
43552025-03-07T22:37:24.114ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
43562025-03-07T22:37:24.114ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
43572025-03-07T22:37:24.114ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 152.458605ms iteration = 8 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
43582025-03-07T22:37:24.132ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
43592025-03-07T22:37:24.132ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "32634cde-c307-4ff7-b763-01455ee9b01b", "content-length": "1", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
43602025-03-07T22:37:24.134ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
43612025-03-07T22:37:24.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
43622025-03-07T22:37:24.136ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 174.353015ms iteration = 8 status = null
43632025-03-07T22:37:24.139ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 177.269843ms iteration = 9 status = {"physical_disks_added":0}
43642025-03-07T22:37:24.167ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 205.02303ms iteration = 8 status = {}
43652025-03-07T22:37:24.210ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
43662025-03-07T22:37:24.210ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
43672025-03-07T22:37:24.210ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "36db63d5-934e-460f-8053-ed4df6e32580", "content-length": "372", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
43682025-03-07T22:37:24.210ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43692025-03-07T22:37:24.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43702025-03-07T22:37:24.213ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
43712025-03-07T22:37:24.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
43722025-03-07T22:37:24.244ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43732025-03-07T22:37:24.246ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
43742025-03-07T22:37:24.246ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
43752025-03-07T22:37:24.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
43762025-03-07T22:37:24.248ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43772025-03-07T22:37:24.249ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43782025-03-07T22:37:24.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43792025-03-07T22:37:24.249ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
43802025-03-07T22:37:24.249ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
43812025-03-07T22:37:24.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
43822025-03-07T22:37:24.251ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
43832025-03-07T22:37:24.251ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43842025-03-07T22:37:24.252ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43852025-03-07T22:37:24.252ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
43862025-03-07T22:37:24.252ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
43872025-03-07T22:37:24.252ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "783044f1-0b85-483e-a184-c828ee84803f", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
43882025-03-07T22:37:24.252ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43892025-03-07T22:37:24.252ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43902025-03-07T22:37:24.254ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43912025-03-07T22:37:24.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43922025-03-07T22:37:24.257ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43932025-03-07T22:37:24.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43942025-03-07T22:37:24.260ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
43952025-03-07T22:37:24.260ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "037ade3b-27ae-49d5-9c1c-db74d79a9d0c", "content-length": "134", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
43962025-03-07T22:37:24.260ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
43972025-03-07T22:37:24.260ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
43982025-03-07T22:37:24.260ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
43992025-03-07T22:37:24.260ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
44002025-03-07T22:37:24.260ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
44012025-03-07T22:37:24.260ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
44022025-03-07T22:37:24.260ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
44032025-03-07T22:37:24.260ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
44042025-03-07T22:37:24.261ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
44052025-03-07T22:37:24.261ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
44062025-03-07T22:37:24.261ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
44072025-03-07T22:37:24.261ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44082025-03-07T22:37:24.261ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
44092025-03-07T22:37:24.261ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
44102025-03-07T22:37:24.261ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44112025-03-07T22:37:24.261ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "840219fa-2251-4238-aa16-71022c2660cb", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
44122025-03-07T22:37:24.263ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
44132025-03-07T22:37:24.263ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cb6417a-6636-4cea-a21c-0177199d149a", "content-length": "2", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
44142025-03-07T22:37:24.263ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
44152025-03-07T22:37:24.263ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "169e284b-2a15-4137-a19c-518d71c554c5", "content-length": "52", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
44162025-03-07T22:37:24.263ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
44172025-03-07T22:37:24.263ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
44182025-03-07T22:37:24.263ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 301.541007ms iteration = 8 status = {}
44192025-03-07T22:37:24.344ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
44202025-03-07T22:37:24.369ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "42a8fba9-2f5b-4ec0-845f-b92c9c294c8a", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
44212025-03-07T22:37:24.369ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
44222025-03-07T22:37:24.369ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
44232025-03-07T22:37:24.370ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dc3ab7d-d386-4786-ad8f-e7214beea2b5", "content-length": "682", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
44242025-03-07T22:37:24.370ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44252025-03-07T22:37:24.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44262025-03-07T22:37:24.370ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7bc215e-f79b-4530-b2fc-f395f5595a03", "content-length": "691", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
44272025-03-07T22:37:24.370ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
44282025-03-07T22:37:24.370ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
44292025-03-07T22:37:24.370ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
44302025-03-07T22:37:24.372ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 410.104351ms iteration = 8 status = {}
44312025-03-07T22:37:24.374ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44322025-03-07T22:37:24.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44332025-03-07T22:37:24.378ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44342025-03-07T22:37:24.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44352025-03-07T22:37:24.380ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44362025-03-07T22:37:24.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44372025-03-07T22:37:24.387ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 424.794112ms iteration = 8 status = {}
44382025-03-07T22:37:24.581ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
44392025-03-07T22:37:24.581ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
44402025-03-07T22:37:24.612ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
44412025-03-07T22:37:24.612ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:42154
44422025-03-07T22:37:24.612ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:42154 req_id = ab611b62-7695-4ac2-ab18-3741e3fd826f roles = RoleSet { roles: {} } uri = /metrics/producers
44432025-03-07T22:37:24.613ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:42154 req_id = ab611b62-7695-4ac2-ab18-3741e3fd826f resource = Database result = Ok(()) uri = /metrics/producers
44442025-03-07T22:37:24.653ZINFOnexus: assigned collector to new producer collector_id = 7b5e9a24-6b39-4a58-a275-8cc5a75195d8 file = nexus/src/app/oximeter.rs:76 producer_id = dfe169a3-7536-4b54-9593-8815d6e81dd4
44452025-03-07T22:37:24.653ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 40248 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:42154 req_id = ab611b62-7695-4ac2-ab18-3741e3fd826f response_code = 201 uri = /metrics/producers
44462025-03-07T22:37:24.653ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "ab611b62-7695-4ac2-ab18-3741e3fd826f", "content-length": "41", "date": "Fri, 07 Mar 2025 22:37:24 GMT"} })
44472025-03-07T22:37:24.655ZDEBGnexus (producer-registration-task): registered with nexus successfully
44482025-03-07T22:37:24.655ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
44492025-03-07T22:37:24.672ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 262 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 9bf41a52-df86-43d7-987f-c08a5f8c76d9 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
44502025-03-07T22:37:33.962ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 23 reason = Timeout
44512025-03-07T22:37:33.962ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 23 reason = Timeout
44522025-03-07T22:37:33.962ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44532025-03-07T22:37:33.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44542025-03-07T22:37:33.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44552025-03-07T22:37:33.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44562025-03-07T22:37:33.993ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44572025-03-07T22:37:33.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44582025-03-07T22:37:33.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44592025-03-07T22:37:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
44602025-03-07T22:37:33.996ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44612025-03-07T22:37:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44622025-03-07T22:37:33.998ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44632025-03-07T22:37:33.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44642025-03-07T22:37:34.000ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44652025-03-07T22:37:34.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
44662025-03-07T22:37:34.002ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44672025-03-07T22:37:34.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44682025-03-07T22:37:34.017ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
44692025-03-07T22:37:34.017ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 54.838017ms iteration = 23 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
44702025-03-07T22:37:34.024ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.361969ms iteration = 23 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
44712025-03-07T22:37:34.663ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = 9d055f8c-2c14-420f-b54d-95ca940362da roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
44722025-03-07T22:37:34.664ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = 9d055f8c-2c14-420f-b54d-95ca940362da resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
44732025-03-07T22:37:34.665ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2105 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = 9d055f8c-2c14-420f-b54d-95ca940362da response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
44742025-03-07T22:37:34.665ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = b0d82423-206a-4f1c-a904-f85f22e8af16 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
44752025-03-07T22:37:34.666ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = b0d82423-206a-4f1c-a904-f85f22e8af16 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
44762025-03-07T22:37:34.668ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2364 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = b0d82423-206a-4f1c-a904-f85f22e8af16 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
44772025-03-07T22:37:34.672ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 275 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = a22c8010-dccb-4e94-9213-b9652c11322b response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
44782025-03-07T22:37:43.962ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 24 reason = Timeout
44792025-03-07T22:37:43.962ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 24 reason = Timeout
44802025-03-07T22:37:43.962ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44812025-03-07T22:37:43.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44822025-03-07T22:37:43.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44832025-03-07T22:37:43.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44842025-03-07T22:37:43.965ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44852025-03-07T22:37:43.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44862025-03-07T22:37:43.966ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.225531ms iteration = 24 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
44872025-03-07T22:37:43.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44882025-03-07T22:37:43.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
44892025-03-07T22:37:43.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44902025-03-07T22:37:43.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44912025-03-07T22:37:43.996ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44922025-03-07T22:37:43.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44932025-03-07T22:37:44.027ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44942025-03-07T22:37:44.029ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
44952025-03-07T22:37:44.029ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44962025-03-07T22:37:44.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44972025-03-07T22:37:44.041ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
44982025-03-07T22:37:44.041ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 78.829467ms iteration = 24 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
44992025-03-07T22:37:44.672ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 266 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = ccffb179-1db8-4eaf-8599-be219646b24f response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
45002025-03-07T22:37:49.663ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = c392f075-2ba3-45d2-936d-eda6743e5b20 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
45012025-03-07T22:37:49.663ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = c392f075-2ba3-45d2-936d-eda6743e5b20 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
45022025-03-07T22:37:49.666ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2975 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = c392f075-2ba3-45d2-936d-eda6743e5b20 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
45032025-03-07T22:37:49.666ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = 7d683e54-1f1a-4e3b-b7a6-28596c4ea92d roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
45042025-03-07T22:37:49.667ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = 7d683e54-1f1a-4e3b-b7a6-28596c4ea92d resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
45052025-03-07T22:37:49.669ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2342 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49063 req_id = 7d683e54-1f1a-4e3b-b7a6-28596c4ea92d response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
45062025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 25 reason = Timeout
45072025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 10 reason = Timeout
45082025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45092025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
45102025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 9 reason = Timeout
45112025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 5 reason = Timeout
45122025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
45132025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 7 reason = Timeout
45142025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 9 reason = Timeout
45152025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
45162025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
45172025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 5 reason = Timeout
45182025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
45192025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 6 reason = Timeout
45202025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 5 reason = Timeout
45212025-03-07T22:37:53.962ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
45222025-03-07T22:37:53.962ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
45232025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.27\u{b5}s iteration = 6 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
45242025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 5 reason = Timeout
45252025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
45262025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
45272025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
45282025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 9 reason = Timeout
45292025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 5 reason = Timeout
45302025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
45312025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 9 reason = Timeout
45322025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 5 reason = Timeout
45332025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
45342025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
45352025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
45362025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 9 reason = Timeout
45372025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
45382025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 7 reason = Timeout
45392025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 5 reason = Timeout
45402025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
45412025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 9 reason = Timeout
45422025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 5 reason = Timeout
45432025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
45442025-03-07T22:37:53.962ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:27:53.962517153 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
45452025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
45462025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 9 reason = Timeout
45472025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
45482025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 9 reason = Timeout
45492025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
45502025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 9 reason = Timeout
45512025-03-07T22:37:53.962ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
45522025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 9 reason = Timeout
45532025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45542025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
45552025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 9 reason = Timeout
45562025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 25 reason = Timeout
45572025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
45582025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45592025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
45602025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
45612025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
45622025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
45632025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
45642025-03-07T22:37:53.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
45652025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
45662025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
45672025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
45682025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
45692025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
45702025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
45712025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
45722025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
45732025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
45742025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
45752025-03-07T22:37:53.963ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1032, tv_nsec: 318357828 } })
45762025-03-07T22:37:53.963ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1032, tv_nsec: 318367818 } })
45772025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45782025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45792025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 9 reason = Timeout
45802025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 9 reason = Timeout
45812025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 5 reason = Timeout
45822025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
45832025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
45842025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 9 reason = Timeout
45852025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
45862025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
45872025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 9 reason = Timeout
45882025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
45892025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 5 reason = Timeout
45902025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
45912025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 9 reason = Timeout
45922025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
45932025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
45942025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
45952025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
45962025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
45972025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
45982025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
45992025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
46002025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
46012025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.215661ms iteration = 5 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
46022025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
46032025-03-07T22:37:53.963ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.356723ms iteration = 5 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
46042025-03-07T22:37:53.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46052025-03-07T22:37:53.965ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
46062025-03-07T22:37:53.965ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46072025-03-07T22:37:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
46082025-03-07T22:37:53.966ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.426713ms iteration = 9 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
46092025-03-07T22:37:53.966ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46102025-03-07T22:37:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
46112025-03-07T22:37:53.968ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
46122025-03-07T22:37:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46132025-03-07T22:37:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
46142025-03-07T22:37:53.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46152025-03-07T22:37:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46162025-03-07T22:37:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
46172025-03-07T22:37:53.970ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46182025-03-07T22:37:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46192025-03-07T22:37:53.971ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1032, tv_nsec: 326810169 } })
46202025-03-07T22:37:53.971ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
46212025-03-07T22:37:53.971ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
46222025-03-07T22:37:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
46232025-03-07T22:37:53.972ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46242025-03-07T22:37:53.973ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46252025-03-07T22:37:53.973ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1032, tv_nsec: 329078861 } })
46262025-03-07T22:37:53.974ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
46272025-03-07T22:37:53.974ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
46282025-03-07T22:37:53.974ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:27:53.962517153 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
46292025-03-07T22:37:53.974ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 11.989855ms iteration = 5 status = {"expiration":"2025-03-07T22:27:53.962517153Z","pruned":[]}
46302025-03-07T22:37:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
46312025-03-07T22:37:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
46322025-03-07T22:37:53.974ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
46332025-03-07T22:37:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
46342025-03-07T22:37:53.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
46352025-03-07T22:37:53.975ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46362025-03-07T22:37:53.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46372025-03-07T22:37:53.981ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46382025-03-07T22:37:53.982ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46392025-03-07T22:37:53.983ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 20.523977ms iteration = 5 status = {"errors":[],"found_port_ok":[]}
46402025-03-07T22:37:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
46412025-03-07T22:37:53.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
46422025-03-07T22:37:53.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
46432025-03-07T22:37:53.984ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
46442025-03-07T22:37:53.984ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46452025-03-07T22:37:53.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46462025-03-07T22:37:53.995ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
46472025-03-07T22:37:53.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46482025-03-07T22:37:53.996ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46492025-03-07T22:37:53.997ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46502025-03-07T22:37:53.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46512025-03-07T22:37:54.000ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46522025-03-07T22:37:54.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46532025-03-07T22:37:54.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
46542025-03-07T22:37:54.009ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
46552025-03-07T22:37:54.009ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
46562025-03-07T22:37:54.009ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
46572025-03-07T22:37:54.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
46582025-03-07T22:37:54.009ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1032, tv_nsec: 365115538 } })
46592025-03-07T22:37:54.010ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
46602025-03-07T22:37:54.012ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46612025-03-07T22:37:54.015ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
46622025-03-07T22:37:54.015ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46632025-03-07T22:37:54.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46642025-03-07T22:37:54.017ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46652025-03-07T22:37:54.017ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46662025-03-07T22:37:54.018ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
46672025-03-07T22:37:54.019ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
46682025-03-07T22:37:54.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
46692025-03-07T22:37:54.020ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
46702025-03-07T22:37:54.020ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
46712025-03-07T22:37:54.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46722025-03-07T22:37:54.021ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 58.423571ms iteration = 9 status = {"errors":[],"garbage_collect_requested":[]}
46732025-03-07T22:37:54.022ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
46742025-03-07T22:37:54.022ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46752025-03-07T22:37:54.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
46762025-03-07T22:37:54.026ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
46772025-03-07T22:37:54.026ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46782025-03-07T22:37:54.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46792025-03-07T22:37:54.029ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 66.846222ms iteration = 25 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
46802025-03-07T22:37:54.029ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46812025-03-07T22:37:54.029ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 66.61851ms iteration = 9 status = {"errors":[],"finish_invoked_ok":[]}
46822025-03-07T22:37:54.030ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
46832025-03-07T22:37:54.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
46842025-03-07T22:37:54.031ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
46852025-03-07T22:37:54.031ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
46862025-03-07T22:37:54.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
46872025-03-07T22:37:54.033ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
46882025-03-07T22:37:54.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
46892025-03-07T22:37:54.036ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46902025-03-07T22:37:54.040ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
46912025-03-07T22:37:54.043ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
46922025-03-07T22:37:54.043ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
46932025-03-07T22:37:54.043ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 79.935858ms iteration = 5 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
46942025-03-07T22:37:54.046ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46952025-03-07T22:37:54.049ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 87.221808ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
46962025-03-07T22:37:54.049ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
46972025-03-07T22:37:54.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
46982025-03-07T22:37:54.051ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
46992025-03-07T22:37:54.051ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
47002025-03-07T22:37:54.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
47012025-03-07T22:37:54.052ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
47022025-03-07T22:37:54.052ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 90.61854ms iteration = 25 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
47032025-03-07T22:37:54.053ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
47042025-03-07T22:37:54.053ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
47052025-03-07T22:37:54.053ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 91.330087ms iteration = 9 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
47062025-03-07T22:37:54.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
47072025-03-07T22:37:54.064ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
47082025-03-07T22:37:54.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
47092025-03-07T22:37:54.065ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 101.863158ms iteration = 9 status = {"errors":[],"requests_created_ok":[]}
47102025-03-07T22:37:54.072ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
47112025-03-07T22:37:54.072ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
47122025-03-07T22:37:54.072ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
47132025-03-07T22:37:54.072ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
47142025-03-07T22:37:54.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
47152025-03-07T22:37:54.073ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
47162025-03-07T22:37:54.073ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
47172025-03-07T22:37:54.073ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 110.661943ms iteration = 5 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
47182025-03-07T22:37:54.073ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
47192025-03-07T22:37:54.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
47202025-03-07T22:37:54.077ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47212025-03-07T22:37:54.078ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
47222025-03-07T22:37:54.078ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47232025-03-07T22:37:54.081ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
47242025-03-07T22:37:54.081ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
47252025-03-07T22:37:54.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
47262025-03-07T22:37:54.082ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
47272025-03-07T22:37:54.082ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
47282025-03-07T22:37:54.082ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
47292025-03-07T22:37:54.083ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
47302025-03-07T22:37:54.083ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
47312025-03-07T22:37:54.083ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
47322025-03-07T22:37:54.083ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 121.558908ms iteration = 9 status = 0
47332025-03-07T22:37:54.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
47342025-03-07T22:37:54.095ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
47352025-03-07T22:37:54.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
47362025-03-07T22:37:54.096ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
47372025-03-07T22:37:54.097ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
47382025-03-07T22:37:54.098ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
47392025-03-07T22:37:54.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
47402025-03-07T22:37:54.098ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
47412025-03-07T22:37:54.099ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47422025-03-07T22:37:54.101ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
47432025-03-07T22:37:54.101ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
47442025-03-07T22:37:54.101ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
47452025-03-07T22:37:54.101ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
47462025-03-07T22:37:54.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
47472025-03-07T22:37:54.102ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
47482025-03-07T22:37:54.102ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
47492025-03-07T22:37:54.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
47502025-03-07T22:37:54.103ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47512025-03-07T22:37:54.108ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
47522025-03-07T22:37:54.108ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
47532025-03-07T22:37:54.108ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
47542025-03-07T22:37:54.130ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 167.140746ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
47552025-03-07T22:37:54.130ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
47562025-03-07T22:37:54.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
47572025-03-07T22:37:54.133ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47582025-03-07T22:37:54.133ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
47592025-03-07T22:37:54.133ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
47602025-03-07T22:37:54.133ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
47612025-03-07T22:37:54.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
47622025-03-07T22:37:54.134ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
47632025-03-07T22:37:54.134ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
47642025-03-07T22:37:54.134ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6719d332-3ab1-476c-b37c-280dc6391d62", "content-length": "26", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} })
47652025-03-07T22:37:54.134ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
47662025-03-07T22:37:54.135ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
47672025-03-07T22:37:54.135ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
47682025-03-07T22:37:54.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
47692025-03-07T22:37:54.135ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
47702025-03-07T22:37:54.135ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
47712025-03-07T22:37:54.135ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8ea03e0-24ad-42c5-ba71-e97b5e4b1e75", "content-length": "26", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} })
47722025-03-07T22:37:54.135ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
47732025-03-07T22:37:54.139ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47742025-03-07T22:37:54.141ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
47752025-03-07T22:37:54.141ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
47762025-03-07T22:37:54.141ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 179.215432ms iteration = 9 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
47772025-03-07T22:37:54.142ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
47782025-03-07T22:37:54.142ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
47792025-03-07T22:37:54.143ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
47802025-03-07T22:37:54.143ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
47812025-03-07T22:37:54.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
47822025-03-07T22:37:54.149ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
47832025-03-07T22:37:54.149ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
47842025-03-07T22:37:54.149ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-07 22:34:17.208516 UTC dns_group = internal generation = 1
47852025-03-07T22:37:54.149ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 186.852385ms iteration = 5 status = {"generation":1}
47862025-03-07T22:37:54.153ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 190.037946ms iteration = 9 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
47872025-03-07T22:37:54.157ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
47882025-03-07T22:37:54.157ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
47892025-03-07T22:37:54.158ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
47902025-03-07T22:37:54.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
47912025-03-07T22:37:54.159ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
47922025-03-07T22:37:54.159ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 197.202894ms iteration = 9 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
47932025-03-07T22:37:54.160ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47942025-03-07T22:37:54.165ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
47952025-03-07T22:37:54.165ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
47962025-03-07T22:37:54.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
47972025-03-07T22:37:54.168ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
47982025-03-07T22:37:54.168ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
47992025-03-07T22:37:54.168ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-07 22:34:17.797135 UTC dns_group = external generation = 2
48002025-03-07T22:37:54.168ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 205.688227ms iteration = 5 status = {"generation":2}
48012025-03-07T22:37:54.190ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 228.019141ms iteration = 10 status = {"physical_disks_added":0}
48022025-03-07T22:37:54.190ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
48032025-03-07T22:37:54.190ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
48042025-03-07T22:37:54.191ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "42569914-57d5-47bb-b788-7a424397db12", "content-length": "26", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} })
48052025-03-07T22:37:54.191ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
48062025-03-07T22:37:54.193ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
48072025-03-07T22:37:54.209ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
48082025-03-07T22:37:54.209ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 247.054524ms iteration = 5 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
48092025-03-07T22:37:54.235ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
48102025-03-07T22:37:54.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
48112025-03-07T22:37:54.237ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48122025-03-07T22:37:54.238ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
48132025-03-07T22:37:54.238ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
48142025-03-07T22:37:54.238ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
48152025-03-07T22:37:54.246ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
48162025-03-07T22:37:54.246ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "d3ba2612-311b-4263-a259-a84f425ed831", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
48172025-03-07T22:37:54.246ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
48182025-03-07T22:37:54.248ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "06336e6d-3e61-47fd-bc08-cbdd489dec6f", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
48192025-03-07T22:37:54.248ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
48202025-03-07T22:37:54.249ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
48212025-03-07T22:37:54.249ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 287.555074ms iteration = 7 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
48222025-03-07T22:37:54.282ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) }
48232025-03-07T22:37:54.282ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 320.185047ms iteration = 5 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
48242025-03-07T22:37:54.308ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
48252025-03-07T22:37:54.308ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
48262025-03-07T22:37:54.309ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "1008c5e2-0e6c-4d5e-b7a4-ebe0e3c240b9", "content-length": "372", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} })
48272025-03-07T22:37:54.309ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48282025-03-07T22:37:54.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48292025-03-07T22:37:54.310ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48302025-03-07T22:37:54.312ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
48312025-03-07T22:37:54.312ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
48322025-03-07T22:37:54.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
48332025-03-07T22:37:54.319ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
48342025-03-07T22:37:54.320ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd9adc5c-4bca-4331-a870-3f88d91c6361", "content-length": "1", "date": "Fri, 07 Mar 2025 22:37:53 GMT"} })
48352025-03-07T22:37:54.321ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
48362025-03-07T22:37:54.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
48372025-03-07T22:37:54.323ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 360.718696ms iteration = 9 status = null
48382025-03-07T22:37:54.353ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
48392025-03-07T22:37:54.353ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
48402025-03-07T22:37:54.354ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "1a58f561-a8a9-4753-9f11-0b0e9e80e5e0", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} })
48412025-03-07T22:37:54.354ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48422025-03-07T22:37:54.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48432025-03-07T22:37:54.357ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48442025-03-07T22:37:54.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48452025-03-07T22:37:54.359ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48462025-03-07T22:37:54.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48472025-03-07T22:37:54.456ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
48482025-03-07T22:37:54.459ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
48492025-03-07T22:37:54.460ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "8451fce0-e731-4e30-8b22-1d036a465171", "content-length": "691", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} })
48502025-03-07T22:37:54.460ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
48512025-03-07T22:37:54.460ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
48522025-03-07T22:37:54.460ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
48532025-03-07T22:37:54.462ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 499.690502ms iteration = 9 status = {}
48542025-03-07T22:37:54.465ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
48552025-03-07T22:37:54.465ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
48562025-03-07T22:37:54.465ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "bbb1c469-8440-468c-bd95-c7767916def7", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} })
48572025-03-07T22:37:54.468ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 505.589848ms iteration = 9 status = {}
48582025-03-07T22:37:54.484ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 522.296129ms iteration = 9 status = {}
48592025-03-07T22:37:54.597ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
48602025-03-07T22:37:54.598ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "5ab7a5fa-c9dc-419b-a974-0884eafa792a", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
48612025-03-07T22:37:54.598ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
48622025-03-07T22:37:54.599ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "15e9f068-0cf1-4a00-90c5-79f6d5354f9c", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
48632025-03-07T22:37:54.599ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
48642025-03-07T22:37:54.601ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "42217f1c-3718-4a1d-ba4c-973566d76536", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
48652025-03-07T22:37:54.601ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
48662025-03-07T22:37:54.602ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
48672025-03-07T22:37:54.602ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 640.267913ms iteration = 7 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
48682025-03-07T22:37:54.640ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48692025-03-07T22:37:54.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48702025-03-07T22:37:54.642ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
48712025-03-07T22:37:54.642ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48722025-03-07T22:37:54.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48732025-03-07T22:37:54.649ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
48742025-03-07T22:37:54.649ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d7a9d7d-26de-40f3-badc-0f2b28130b78", "content-length": "134", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} })
48752025-03-07T22:37:54.649ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
48762025-03-07T22:37:54.649ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
48772025-03-07T22:37:54.650ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
48782025-03-07T22:37:54.650ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
48792025-03-07T22:37:54.650ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
48802025-03-07T22:37:54.650ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
48812025-03-07T22:37:54.650ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
48822025-03-07T22:37:54.650ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
48832025-03-07T22:37:54.650ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
48842025-03-07T22:37:54.650ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
48852025-03-07T22:37:54.650ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
48862025-03-07T22:37:54.650ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48872025-03-07T22:37:54.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48882025-03-07T22:37:54.653ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
48892025-03-07T22:37:54.653ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "1619f5ac-eebe-4a76-8f67-e21ee51bd61e", "content-length": "2", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} })
48902025-03-07T22:37:54.653ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
48912025-03-07T22:37:54.653ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbc35218-232d-4d9e-8714-84acb17948e0", "content-length": "52", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} })
48922025-03-07T22:37:54.653ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
48932025-03-07T22:37:54.653ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
48942025-03-07T22:37:54.672ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 270 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 91ec9b0a-13d0-4abd-bda9-1c5420561239 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
48952025-03-07T22:37:54.747ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "81e906f0-4949-4882-9c1a-6940ce9ae6c1", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} })
48962025-03-07T22:37:54.747ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
48972025-03-07T22:37:54.747ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
48982025-03-07T22:37:54.747ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "991761d2-3f38-4528-879e-71858710bf2c", "content-length": "682", "date": "Fri, 07 Mar 2025 22:37:54 GMT"} })
48992025-03-07T22:37:54.747ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49002025-03-07T22:37:54.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49012025-03-07T22:37:54.752ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49022025-03-07T22:37:54.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49032025-03-07T22:37:54.756ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49042025-03-07T22:37:54.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49052025-03-07T22:37:54.758ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49062025-03-07T22:37:54.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49072025-03-07T22:37:54.765ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 803.37224ms iteration = 9 status = {}
49082025-03-07T22:38:03.962ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 26 reason = Timeout
49092025-03-07T22:38:03.962ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49102025-03-07T22:38:03.962ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 26 reason = Timeout
49112025-03-07T22:38:03.962ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49122025-03-07T22:38:03.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49132025-03-07T22:38:03.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49142025-03-07T22:38:03.964ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49152025-03-07T22:38:03.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49162025-03-07T22:38:03.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49172025-03-07T22:38:03.966ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.695965ms iteration = 26 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
49182025-03-07T22:38:03.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
49192025-03-07T22:38:03.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49202025-03-07T22:38:03.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49212025-03-07T22:38:03.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49222025-03-07T22:38:03.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49232025-03-07T22:38:03.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49242025-03-07T22:38:03.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
49252025-03-07T22:38:03.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49262025-03-07T22:38:03.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49272025-03-07T22:38:03.981ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
49282025-03-07T22:38:03.981ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.412767ms iteration = 26 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
49292025-03-07T22:38:04.672ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 262 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 7dfbb941-eda9-4907-b6c5-8d34ee9d4f6b response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
49302025-03-07T22:38:13.963ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 27 reason = Timeout
49312025-03-07T22:38:13.963ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 27 reason = Timeout
49322025-03-07T22:38:13.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49332025-03-07T22:38:13.963ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49342025-03-07T22:38:13.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49352025-03-07T22:38:13.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49362025-03-07T22:38:13.964ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49372025-03-07T22:38:13.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49382025-03-07T22:38:13.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49392025-03-07T22:38:13.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
49402025-03-07T22:38:13.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49412025-03-07T22:38:13.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49422025-03-07T22:38:13.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49432025-03-07T22:38:13.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49442025-03-07T22:38:13.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49452025-03-07T22:38:13.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
49462025-03-07T22:38:13.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49472025-03-07T22:38:13.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49482025-03-07T22:38:13.985ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
49492025-03-07T22:38:13.985ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.578087ms iteration = 27 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
49502025-03-07T22:38:13.992ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 29.451653ms iteration = 27 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
49512025-03-07T22:38:14.673ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 296 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 1149bbe4-b186-4b83-b26a-6d0675fb9d28 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
49522025-03-07T22:38:23.963ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 28 reason = Timeout
49532025-03-07T22:38:23.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49542025-03-07T22:38:23.963ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 28 reason = Timeout
49552025-03-07T22:38:23.963ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49562025-03-07T22:38:23.963ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 10 reason = Timeout
49572025-03-07T22:38:23.963ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 10 reason = Timeout
49582025-03-07T22:38:23.963ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49592025-03-07T22:38:23.963ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 10 reason = Timeout
49602025-03-07T22:38:23.963ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
49612025-03-07T22:38:23.963ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
49622025-03-07T22:38:23.963ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49632025-03-07T22:38:23.963ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 10 reason = Timeout
49642025-03-07T22:38:23.963ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
49652025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 10 reason = Timeout
49662025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
49672025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
49682025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 10 reason = Timeout
49692025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
49702025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 11 reason = Timeout
49712025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
49722025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 10 reason = Timeout
49732025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
49742025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 10 reason = Timeout
49752025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
49762025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 10 reason = Timeout
49772025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 10 reason = Timeout
49782025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49792025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
49802025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 10 reason = Timeout
49812025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 10 reason = Timeout
49822025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 10 reason = Timeout
49832025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
49842025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 10 reason = Timeout
49852025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
49862025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
49872025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 10 reason = Timeout
49882025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
49892025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 10 reason = Timeout
49902025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
49912025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49922025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49932025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
49942025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49952025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
49962025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
49972025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49982025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
49992025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
50002025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50012025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50022025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
50032025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
50042025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
50052025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
50062025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
50072025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
50082025-03-07T22:38:23.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
50092025-03-07T22:38:23.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50102025-03-07T22:38:23.966ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50112025-03-07T22:38:23.967ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50122025-03-07T22:38:23.967ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
50132025-03-07T22:38:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
50142025-03-07T22:38:23.967ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.968848ms iteration = 10 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
50152025-03-07T22:38:23.968ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50162025-03-07T22:38:23.969ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
50172025-03-07T22:38:23.969ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50182025-03-07T22:38:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50192025-03-07T22:38:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
50202025-03-07T22:38:23.969ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
50212025-03-07T22:38:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
50222025-03-07T22:38:23.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50232025-03-07T22:38:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
50242025-03-07T22:38:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50252025-03-07T22:38:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
50262025-03-07T22:38:23.970ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
50272025-03-07T22:38:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
50282025-03-07T22:38:23.971ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50292025-03-07T22:38:23.971ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50302025-03-07T22:38:23.973ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50312025-03-07T22:38:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
50322025-03-07T22:38:23.973ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50332025-03-07T22:38:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50342025-03-07T22:38:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
50352025-03-07T22:38:23.973ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50362025-03-07T22:38:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50372025-03-07T22:38:23.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
50382025-03-07T22:38:23.974ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50392025-03-07T22:38:23.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50402025-03-07T22:38:23.976ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1062, tv_nsec: 331190941 } })
50412025-03-07T22:38:23.976ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
50422025-03-07T22:38:23.976ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1062, tv_nsec: 331578995 } })
50432025-03-07T22:38:23.977ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
50442025-03-07T22:38:23.978ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50452025-03-07T22:38:23.980ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
50462025-03-07T22:38:23.980ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50472025-03-07T22:38:23.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50482025-03-07T22:38:23.980ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50492025-03-07T22:38:23.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50502025-03-07T22:38:23.980ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50512025-03-07T22:38:23.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50522025-03-07T22:38:23.981ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 17.718341ms iteration = 10 status = {"errors":[],"requests_created_ok":[]}
50532025-03-07T22:38:23.983ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1062, tv_nsec: 338154499 } })
50542025-03-07T22:38:23.983ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
50552025-03-07T22:38:23.985ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50562025-03-07T22:38:23.985ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
50572025-03-07T22:38:23.985ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 22.013412ms iteration = 10 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
50582025-03-07T22:38:23.986ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50592025-03-07T22:38:23.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50602025-03-07T22:38:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
50612025-03-07T22:38:23.988ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50622025-03-07T22:38:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50632025-03-07T22:38:23.992ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 28.512744ms iteration = 11 status = {"physical_disks_added":0}
50642025-03-07T22:38:24.007ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
50652025-03-07T22:38:24.007ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
50662025-03-07T22:38:24.007ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
50672025-03-07T22:38:24.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
50682025-03-07T22:38:24.008ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
50692025-03-07T22:38:24.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
50702025-03-07T22:38:24.010ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 47.007242ms iteration = 28 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
50712025-03-07T22:38:24.011ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50722025-03-07T22:38:24.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50732025-03-07T22:38:24.014ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50742025-03-07T22:38:24.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50752025-03-07T22:38:24.014ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
50762025-03-07T22:38:24.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
50772025-03-07T22:38:24.016ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50782025-03-07T22:38:24.017ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
50792025-03-07T22:38:24.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
50802025-03-07T22:38:24.019ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 55.362022ms iteration = 10 status = {"errors":[],"garbage_collect_requested":[]}
50812025-03-07T22:38:24.019ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
50822025-03-07T22:38:24.019ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50832025-03-07T22:38:24.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50842025-03-07T22:38:24.020ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 56.127149ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
50852025-03-07T22:38:24.020ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
50862025-03-07T22:38:24.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
50872025-03-07T22:38:24.021ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 57.880376ms iteration = 10 status = 0
50882025-03-07T22:38:24.022ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
50892025-03-07T22:38:24.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
50902025-03-07T22:38:24.022ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 58.763185ms iteration = 10 status = {"errors":[],"finish_invoked_ok":[]}
50912025-03-07T22:38:24.023ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50922025-03-07T22:38:24.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
50932025-03-07T22:38:24.024ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
50942025-03-07T22:38:24.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
50952025-03-07T22:38:24.028ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50962025-03-07T22:38:24.031ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
50972025-03-07T22:38:24.031ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50982025-03-07T22:38:24.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50992025-03-07T22:38:24.042ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
51002025-03-07T22:38:24.042ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 78.887248ms iteration = 28 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
51012025-03-07T22:38:24.046ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
51022025-03-07T22:38:24.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
51032025-03-07T22:38:24.050ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
51042025-03-07T22:38:24.050ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
51052025-03-07T22:38:24.050ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
51062025-03-07T22:38:24.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
51072025-03-07T22:38:24.065ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 101.468765ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
51082025-03-07T22:38:24.065ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
51092025-03-07T22:38:24.065ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
51102025-03-07T22:38:24.066ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "528c3957-d15f-44c3-a838-3d019cf1a92e", "content-length": "26", "date": "Fri, 07 Mar 2025 22:38:24 GMT"} })
51112025-03-07T22:38:24.066ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
51122025-03-07T22:38:24.067ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
51132025-03-07T22:38:24.069ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
51142025-03-07T22:38:24.069ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 105.809686ms iteration = 10 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
51152025-03-07T22:38:24.078ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
51162025-03-07T22:38:24.078ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
51172025-03-07T22:38:24.078ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "63547803-becc-4c51-b1c3-a82d2b2bde3a", "content-length": "26", "date": "Fri, 07 Mar 2025 22:38:24 GMT"} })
51182025-03-07T22:38:24.078ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
51192025-03-07T22:38:24.080ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
51202025-03-07T22:38:24.084ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
51212025-03-07T22:38:24.084ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
51222025-03-07T22:38:24.084ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c891164-2f8c-4fae-8cf4-6ad1cc6c95cc", "content-length": "26", "date": "Fri, 07 Mar 2025 22:38:24 GMT"} })
51232025-03-07T22:38:24.084ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
51242025-03-07T22:38:24.086ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
51252025-03-07T22:38:24.087ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 122.947891ms iteration = 10 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
51262025-03-07T22:38:24.096ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
51272025-03-07T22:38:24.096ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
51282025-03-07T22:38:24.096ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 132.738055ms iteration = 10 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
51292025-03-07T22:38:24.140ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
51302025-03-07T22:38:24.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
51312025-03-07T22:38:24.142ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51322025-03-07T22:38:24.144ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
51332025-03-07T22:38:24.144ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
51342025-03-07T22:38:24.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
51352025-03-07T22:38:24.180ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
51362025-03-07T22:38:24.180ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
51372025-03-07T22:38:24.180ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "02317274-bd17-4c24-b1c2-cc1a8687071b", "content-length": "372", "date": "Fri, 07 Mar 2025 22:38:24 GMT"} })
51382025-03-07T22:38:24.180ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51392025-03-07T22:38:24.180ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51402025-03-07T22:38:24.181ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51412025-03-07T22:38:24.183ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
51422025-03-07T22:38:24.183ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
51432025-03-07T22:38:24.183ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
51442025-03-07T22:38:24.196ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
51452025-03-07T22:38:24.196ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
51462025-03-07T22:38:24.196ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "2801cb60-ef2e-4916-a1ed-c3b8c3704c15", "date": "Fri, 07 Mar 2025 22:38:24 GMT"} })
51472025-03-07T22:38:24.196ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51482025-03-07T22:38:24.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51492025-03-07T22:38:24.197ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 233.309472ms iteration = 10 status = {}
51502025-03-07T22:38:24.198ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51512025-03-07T22:38:24.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51522025-03-07T22:38:24.267ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
51532025-03-07T22:38:24.267ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "204cf212-303d-47d5-bcd1-5c46a2a87c67", "content-length": "1", "date": "Fri, 07 Mar 2025 22:38:23 GMT"} })
51542025-03-07T22:38:24.269ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
51552025-03-07T22:38:24.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
51562025-03-07T22:38:24.274ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
51572025-03-07T22:38:24.274ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d04aa39-3257-480b-8fec-82fde4edf29a", "content-length": "691", "date": "Fri, 07 Mar 2025 22:38:24 GMT"} })
51582025-03-07T22:38:24.274ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
51592025-03-07T22:38:24.274ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
51602025-03-07T22:38:24.274ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
51612025-03-07T22:38:24.276ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 312.189639ms iteration = 10 status = {}
51622025-03-07T22:38:24.277ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51632025-03-07T22:38:24.277ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 313.901196ms iteration = 10 status = null
51642025-03-07T22:38:24.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51652025-03-07T22:38:24.281ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
51662025-03-07T22:38:24.281ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
51672025-03-07T22:38:24.281ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "16384cb5-451b-47e3-b761-34bb83b689c5", "date": "Fri, 07 Mar 2025 22:38:24 GMT"} })
51682025-03-07T22:38:24.283ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 319.809953ms iteration = 10 status = {}
51692025-03-07T22:38:24.339ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51702025-03-07T22:38:24.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51712025-03-07T22:38:24.341ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
51722025-03-07T22:38:24.341ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51732025-03-07T22:38:24.341ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51742025-03-07T22:38:24.349ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
51752025-03-07T22:38:24.349ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "87220ffb-8104-4c18-80d3-33f029e84e31", "content-length": "134", "date": "Fri, 07 Mar 2025 22:38:23 GMT"} })
51762025-03-07T22:38:24.349ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
51772025-03-07T22:38:24.349ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
51782025-03-07T22:38:24.349ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
51792025-03-07T22:38:24.349ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
51802025-03-07T22:38:24.349ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
51812025-03-07T22:38:24.349ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
51822025-03-07T22:38:24.349ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
51832025-03-07T22:38:24.349ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
51842025-03-07T22:38:24.349ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
51852025-03-07T22:38:24.349ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
51862025-03-07T22:38:24.350ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
51872025-03-07T22:38:24.350ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51882025-03-07T22:38:24.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51892025-03-07T22:38:24.352ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
51902025-03-07T22:38:24.352ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "b212c96c-fddb-4bcf-8762-a3f54e2b9790", "content-length": "2", "date": "Fri, 07 Mar 2025 22:38:23 GMT"} })
51912025-03-07T22:38:24.352ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
51922025-03-07T22:38:24.352ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "1769679d-afae-46bd-ad4e-a69c0f7100bf", "content-length": "52", "date": "Fri, 07 Mar 2025 22:38:23 GMT"} })
51932025-03-07T22:38:24.352ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
51942025-03-07T22:38:24.352ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
51952025-03-07T22:38:24.468ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "2f55998b-ef8c-419c-8845-a48113a80e9e", "date": "Fri, 07 Mar 2025 22:38:24 GMT"} })
51962025-03-07T22:38:24.468ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
51972025-03-07T22:38:24.468ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
51982025-03-07T22:38:24.468ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d3744e2-dfd7-4edf-b9da-bc3538e113db", "content-length": "682", "date": "Fri, 07 Mar 2025 22:38:24 GMT"} })
51992025-03-07T22:38:24.468ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52002025-03-07T22:38:24.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52012025-03-07T22:38:24.477ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52022025-03-07T22:38:24.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52032025-03-07T22:38:24.480ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52042025-03-07T22:38:24.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52052025-03-07T22:38:24.483ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52062025-03-07T22:38:24.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52072025-03-07T22:38:24.490ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 526.371379ms iteration = 10 status = {}
52082025-03-07T22:38:24.673ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 289 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = d8474196-d1e7-4ff8-81bc-5f3c56871e8c response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
52092025-03-07T22:38:33.963ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
52102025-03-07T22:38:33.964ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 29 reason = Timeout
52112025-03-07T22:38:33.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52122025-03-07T22:38:33.964ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52132025-03-07T22:38:33.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52142025-03-07T22:38:33.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52152025-03-07T22:38:33.966ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52162025-03-07T22:38:33.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52172025-03-07T22:38:33.968ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.097499ms iteration = 29 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
52182025-03-07T22:38:33.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52192025-03-07T22:38:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
52202025-03-07T22:38:33.996ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52212025-03-07T22:38:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52222025-03-07T22:38:33.998ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52232025-03-07T22:38:33.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52242025-03-07T22:38:34.000ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52252025-03-07T22:38:34.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
52262025-03-07T22:38:34.002ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52272025-03-07T22:38:34.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52282025-03-07T22:38:34.013ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
52292025-03-07T22:38:34.013ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 49.666688ms iteration = 29 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
52302025-03-07T22:38:34.673ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 296 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 6e95d74a-5b27-4de9-bc14-9e052a13c5f1 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
52312025-03-07T22:38:43.963ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 30 reason = Timeout
52322025-03-07T22:38:43.963ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52332025-03-07T22:38:43.963ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
52342025-03-07T22:38:43.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52352025-03-07T22:38:43.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52362025-03-07T22:38:43.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52372025-03-07T22:38:43.965ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52382025-03-07T22:38:43.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52392025-03-07T22:38:43.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52402025-03-07T22:38:43.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
52412025-03-07T22:38:43.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52422025-03-07T22:38:43.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52432025-03-07T22:38:43.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52442025-03-07T22:38:43.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52452025-03-07T22:38:43.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52462025-03-07T22:38:43.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
52472025-03-07T22:38:43.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52482025-03-07T22:38:43.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52492025-03-07T22:38:43.984ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
52502025-03-07T22:38:43.984ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.220404ms iteration = 30 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
52512025-03-07T22:38:43.993ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 29.973948ms iteration = 30 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
52522025-03-07T22:38:44.673ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 287 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 3ae3c6a4-9c00-4119-8273-96858ab99298 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
52532025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
52542025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
52552025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 11 reason = Timeout
52562025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
52572025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
52582025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
52592025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
52602025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
52612025-03-07T22:38:53.963ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
52622025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 8 reason = Timeout
52632025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 21.8\u{b5}s iteration = 7 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
52642025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
52652025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 11 reason = Timeout
52662025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
52672025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 2 reason = Timeout
52682025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52692025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
52702025-03-07T22:38:53.963ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
52712025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
52722025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 11 reason = Timeout
52732025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52742025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
52752025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
52762025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 12 reason = Timeout
52772025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
52782025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
52792025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
52802025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
52812025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
52822025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
52832025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 3 reason = Timeout
52842025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 6 reason = Timeout
52852025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
52862025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
52872025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
52882025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
52892025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 11 reason = Timeout
52902025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 31 reason = Timeout
52912025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
52922025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
52932025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52942025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 11 reason = Timeout
52952025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52962025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
52972025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 11 reason = Timeout
52982025-03-07T22:38:53.963ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
52992025-03-07T22:38:53.963ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53002025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 6 reason = Timeout
53012025-03-07T22:38:53.964ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:28:53.963919535 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
53022025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
53032025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 8 reason = Timeout
53042025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
53052025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
53062025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
53072025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 6 reason = Timeout
53082025-03-07T22:38:53.964ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
53092025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
53102025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
53112025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
53122025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53132025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
53142025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
53152025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53162025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
53172025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
53182025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
53192025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 11 reason = Timeout
53202025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 6 reason = Timeout
53212025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 11 reason = Timeout
53222025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
53232025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
53242025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 2 reason = Timeout
53252025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
53262025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
53272025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 11 reason = Timeout
53282025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
53292025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 11 reason = Timeout
53302025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
53312025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
53322025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 6 reason = Timeout
53332025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
53342025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 11 reason = Timeout
53352025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
53362025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
53372025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
53382025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53392025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
53402025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
53412025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
53422025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53432025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
53442025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
53452025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
53462025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
53472025-03-07T22:38:53.964ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1092, tv_nsec: 318681911 } })
53482025-03-07T22:38:53.964ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1092, tv_nsec: 318688731 } })
53492025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
53502025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
53512025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
53522025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
53532025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
53542025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
53552025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
53562025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
53572025-03-07T22:38:53.964ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
53582025-03-07T22:38:53.965ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
53592025-03-07T22:38:53.965ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.289692ms iteration = 6 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
53602025-03-07T22:38:53.965ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
53612025-03-07T22:38:53.965ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.09363ms iteration = 6 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
53622025-03-07T22:38:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
53632025-03-07T22:38:53.966ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53642025-03-07T22:38:53.967ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53652025-03-07T22:38:53.967ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.574994ms iteration = 11 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
53662025-03-07T22:38:53.968ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53672025-03-07T22:38:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53682025-03-07T22:38:53.968ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53692025-03-07T22:38:53.968ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
53702025-03-07T22:38:53.968ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
53712025-03-07T22:38:53.968ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
53722025-03-07T22:38:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
53732025-03-07T22:38:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
53742025-03-07T22:38:53.969ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
53752025-03-07T22:38:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
53762025-03-07T22:38:53.969ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
53772025-03-07T22:38:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
53782025-03-07T22:38:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
53792025-03-07T22:38:53.970ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
53802025-03-07T22:38:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53812025-03-07T22:38:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
53822025-03-07T22:38:53.970ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
53832025-03-07T22:38:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
53842025-03-07T22:38:53.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53852025-03-07T22:38:53.972ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
53862025-03-07T22:38:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
53872025-03-07T22:38:53.972ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53882025-03-07T22:38:53.973ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53892025-03-07T22:38:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
53902025-03-07T22:38:53.974ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
53912025-03-07T22:38:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
53922025-03-07T22:38:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
53932025-03-07T22:38:53.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
53942025-03-07T22:38:53.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
53952025-03-07T22:38:53.975ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
53962025-03-07T22:38:53.975ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53972025-03-07T22:38:53.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53982025-03-07T22:38:53.977ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1092, tv_nsec: 331479164 } })
53992025-03-07T22:38:53.977ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
54002025-03-07T22:38:53.978ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54012025-03-07T22:38:53.978ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
54022025-03-07T22:38:53.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
54032025-03-07T22:38:53.979ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1092, tv_nsec: 333172720 } })
54042025-03-07T22:38:53.979ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
54052025-03-07T22:38:53.980ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
54062025-03-07T22:38:53.980ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54072025-03-07T22:38:53.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54082025-03-07T22:38:53.982ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54092025-03-07T22:38:53.982ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:28:53.963919535 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
54102025-03-07T22:38:53.982ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 18.244786ms iteration = 6 status = {"expiration":"2025-03-07T22:28:53.963919535Z","pruned":[]}
54112025-03-07T22:38:53.983ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
54122025-03-07T22:38:53.983ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
54132025-03-07T22:38:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
54142025-03-07T22:38:53.985ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 21.157114ms iteration = 6 status = {"errors":[],"found_port_ok":[]}
54152025-03-07T22:38:53.985ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 21.481357ms iteration = 11 status = {"errors":[],"requests_created_ok":[]}
54162025-03-07T22:38:53.987ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54172025-03-07T22:38:53.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54182025-03-07T22:38:53.989ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
54192025-03-07T22:38:53.989ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
54202025-03-07T22:38:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
54212025-03-07T22:38:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
54222025-03-07T22:38:53.990ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
54232025-03-07T22:38:53.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
54242025-03-07T22:38:53.991ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1092, tv_nsec: 345138435 } })
54252025-03-07T22:38:53.991ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
54262025-03-07T22:38:53.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54272025-03-07T22:38:53.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54282025-03-07T22:38:54.018ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54292025-03-07T22:38:54.019ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54302025-03-07T22:38:54.019ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
54312025-03-07T22:38:54.019ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
54322025-03-07T22:38:54.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
54332025-03-07T22:38:54.021ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
54342025-03-07T22:38:54.021ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
54352025-03-07T22:38:54.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
54362025-03-07T22:38:54.022ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54372025-03-07T22:38:54.022ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54382025-03-07T22:38:54.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54392025-03-07T22:38:54.024ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54402025-03-07T22:38:54.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
54412025-03-07T22:38:54.024ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
54422025-03-07T22:38:54.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
54432025-03-07T22:38:54.025ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
54442025-03-07T22:38:54.025ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
54452025-03-07T22:38:54.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
54462025-03-07T22:38:54.026ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
54472025-03-07T22:38:54.026ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
54482025-03-07T22:38:54.026ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 62.44593ms iteration = 6 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
54492025-03-07T22:38:54.027ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
54502025-03-07T22:38:54.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
54512025-03-07T22:38:54.028ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54522025-03-07T22:38:54.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54532025-03-07T22:38:54.031ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
54542025-03-07T22:38:54.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
54552025-03-07T22:38:54.031ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54562025-03-07T22:38:54.032ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
54572025-03-07T22:38:54.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
54582025-03-07T22:38:54.033ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
54592025-03-07T22:38:54.033ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
54602025-03-07T22:38:54.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
54612025-03-07T22:38:54.034ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
54622025-03-07T22:38:54.034ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
54632025-03-07T22:38:54.034ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
54642025-03-07T22:38:54.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
54652025-03-07T22:38:54.035ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 71.166094ms iteration = 11 status = {"errors":[],"garbage_collect_requested":[]}
54662025-03-07T22:38:54.035ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54672025-03-07T22:38:54.035ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
54682025-03-07T22:38:54.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
54692025-03-07T22:38:54.036ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54702025-03-07T22:38:54.036ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54712025-03-07T22:38:54.036ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 72.320355ms iteration = 11 status = {"errors":[],"finish_invoked_ok":[]}
54722025-03-07T22:38:54.037ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
54732025-03-07T22:38:54.037ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
54742025-03-07T22:38:54.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
54752025-03-07T22:38:54.039ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 76.00132ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
54762025-03-07T22:38:54.039ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54772025-03-07T22:38:54.040ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54782025-03-07T22:38:54.040ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
54792025-03-07T22:38:54.040ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
54802025-03-07T22:38:54.040ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54812025-03-07T22:38:54.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
54822025-03-07T22:38:54.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54832025-03-07T22:38:54.041ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
54842025-03-07T22:38:54.042ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
54852025-03-07T22:38:54.042ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 78.666845ms iteration = 11 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
54862025-03-07T22:38:54.042ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
54872025-03-07T22:38:54.042ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
54882025-03-07T22:38:54.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
54892025-03-07T22:38:54.044ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54902025-03-07T22:38:54.044ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
54912025-03-07T22:38:54.044ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
54922025-03-07T22:38:54.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54932025-03-07T22:38:54.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
54942025-03-07T22:38:54.045ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
54952025-03-07T22:38:54.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
54962025-03-07T22:38:54.046ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54972025-03-07T22:38:54.049ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 85.292759ms iteration = 31 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
54982025-03-07T22:38:54.051ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
54992025-03-07T22:38:54.053ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
55002025-03-07T22:38:54.053ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
55012025-03-07T22:38:54.053ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
55022025-03-07T22:38:54.053ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
55032025-03-07T22:38:54.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
55042025-03-07T22:38:54.059ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
55052025-03-07T22:38:54.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
55062025-03-07T22:38:54.064ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
55072025-03-07T22:38:54.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
55082025-03-07T22:38:54.094ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
55092025-03-07T22:38:54.094ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
55102025-03-07T22:38:54.094ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 130.932018ms iteration = 6 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
55112025-03-07T22:38:54.096ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
55122025-03-07T22:38:54.096ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
55132025-03-07T22:38:54.101ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 137.236969ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
55142025-03-07T22:38:54.101ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
55152025-03-07T22:38:54.101ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
55162025-03-07T22:38:54.101ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
55172025-03-07T22:38:54.101ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
55182025-03-07T22:38:54.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
55192025-03-07T22:38:54.103ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
55202025-03-07T22:38:54.103ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
55212025-03-07T22:38:54.103ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 139.379369ms iteration = 11 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
55222025-03-07T22:38:54.121ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
55232025-03-07T22:38:54.198ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
55242025-03-07T22:38:54.198ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
55252025-03-07T22:38:54.198ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
55262025-03-07T22:38:54.198ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
55272025-03-07T22:38:54.198ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "45935f9d-5fd5-4be8-a8cc-26d376d229f6", "content-length": "26", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} })
55282025-03-07T22:38:54.198ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
55292025-03-07T22:38:54.198ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "480f70cb-6cfa-4738-b496-e74a3b765868", "content-length": "26", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} })
55302025-03-07T22:38:54.198ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
55312025-03-07T22:38:54.199ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 235.968627ms iteration = 12 status = {"physical_disks_added":0}
55322025-03-07T22:38:54.201ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
55332025-03-07T22:38:54.202ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
55342025-03-07T22:38:54.231ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
55352025-03-07T22:38:54.243ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
55362025-03-07T22:38:54.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
55372025-03-07T22:38:54.248ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55382025-03-07T22:38:54.249ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
55392025-03-07T22:38:54.249ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
55402025-03-07T22:38:54.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
55412025-03-07T22:38:54.261ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
55422025-03-07T22:38:54.261ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 298.101474ms iteration = 31 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
55432025-03-07T22:38:54.276ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55442025-03-07T22:38:54.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
55452025-03-07T22:38:54.278ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
55462025-03-07T22:38:54.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
55472025-03-07T22:38:54.281ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 316.742974ms iteration = 11 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
55482025-03-07T22:38:54.283ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55492025-03-07T22:38:54.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55502025-03-07T22:38:54.292ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
55512025-03-07T22:38:54.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
55522025-03-07T22:38:54.306ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
55532025-03-07T22:38:54.306ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "15d25bed-52e5-42bb-aec2-6631876de859", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
55542025-03-07T22:38:54.306ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
55552025-03-07T22:38:54.307ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
55562025-03-07T22:38:54.307ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
55572025-03-07T22:38:54.307ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "75e64abd-de0d-4c87-b51b-34f49741c556", "content-length": "26", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} })
55582025-03-07T22:38:54.307ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
55592025-03-07T22:38:54.308ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "6555cf9f-1026-433e-a967-7cfcae3f4be3", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
55602025-03-07T22:38:54.308ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
55612025-03-07T22:38:54.311ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
55622025-03-07T22:38:54.311ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
55632025-03-07T22:38:54.311ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 348.057945ms iteration = 8 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
55642025-03-07T22:38:54.365ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
55652025-03-07T22:38:54.365ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a3c0445-1f54-4f83-9fec-5cf96ea67406", "content-length": "1", "date": "Fri, 07 Mar 2025 22:38:53 GMT"} })
55662025-03-07T22:38:54.366ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
55672025-03-07T22:38:54.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
55682025-03-07T22:38:54.387ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
55692025-03-07T22:38:54.387ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
55702025-03-07T22:38:54.387ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce605bb3-3165-478d-8e0d-731bbbb17065", "content-length": "372", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} })
55712025-03-07T22:38:54.387ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
55722025-03-07T22:38:54.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
55732025-03-07T22:38:54.389ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
55742025-03-07T22:38:54.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
55752025-03-07T22:38:54.390ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55762025-03-07T22:38:54.392ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
55772025-03-07T22:38:54.392ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
55782025-03-07T22:38:54.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
55792025-03-07T22:38:54.396ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 432.85894ms iteration = 11 status = null
55802025-03-07T22:38:54.396ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55812025-03-07T22:38:54.398ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
55822025-03-07T22:38:54.401ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55832025-03-07T22:38:54.403ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
55842025-03-07T22:38:54.403ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55852025-03-07T22:38:54.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55862025-03-07T22:38:54.404ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
55872025-03-07T22:38:54.404ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
55882025-03-07T22:38:54.404ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
55892025-03-07T22:38:54.409ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 446.100757ms iteration = 11 status = 0
55902025-03-07T22:38:54.426ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 462.542024ms iteration = 11 status = {}
55912025-03-07T22:38:54.430ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55922025-03-07T22:38:54.433ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
55932025-03-07T22:38:54.433ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
55942025-03-07T22:38:54.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
55952025-03-07T22:38:54.444ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55962025-03-07T22:38:54.447ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
55972025-03-07T22:38:54.447ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 483.697849ms iteration = 11 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
55982025-03-07T22:38:54.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
55992025-03-07T22:38:54.448ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
56002025-03-07T22:38:54.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
56012025-03-07T22:38:54.452ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56022025-03-07T22:38:54.457ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
56032025-03-07T22:38:54.457ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
56042025-03-07T22:38:54.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
56052025-03-07T22:38:54.460ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56062025-03-07T22:38:54.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
56072025-03-07T22:38:54.468ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
56082025-03-07T22:38:54.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
56092025-03-07T22:38:54.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56102025-03-07T22:38:54.471ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
56112025-03-07T22:38:54.471ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
56122025-03-07T22:38:54.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
56132025-03-07T22:38:54.475ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
56142025-03-07T22:38:54.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
56152025-03-07T22:38:54.481ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
56162025-03-07T22:38:54.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
56172025-03-07T22:38:54.485ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56182025-03-07T22:38:54.485ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
56192025-03-07T22:38:54.485ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
56202025-03-07T22:38:54.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
56212025-03-07T22:38:54.486ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7743fe8-3c70-4dd7-b3e8-2698e001e158", "content-length": "2", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} })
56222025-03-07T22:38:54.486ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
56232025-03-07T22:38:54.486ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
56242025-03-07T22:38:54.487ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "55005848-4742-40e0-936c-a37a98387cf1", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} })
56252025-03-07T22:38:54.487ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56262025-03-07T22:38:54.487ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
56272025-03-07T22:38:54.487ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
56282025-03-07T22:38:54.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56292025-03-07T22:38:54.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
56302025-03-07T22:38:54.488ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:615 sled = http://[fd00:1122:3344:101::1]:12345
56312025-03-07T22:38:54.490ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56322025-03-07T22:38:54.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56332025-03-07T22:38:54.491ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 527.305417ms iteration = 2 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"73b218dd-7dbc-4345-acb1-fe69ced4a7f7","time":"2025-03-07T22:38:54.488827469Z"}]}
56342025-03-07T22:38:54.494ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56352025-03-07T22:38:54.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56362025-03-07T22:38:54.500ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
56372025-03-07T22:38:54.500ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
56382025-03-07T22:38:54.500ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "4b9def49-0d5a-4b90-ba0c-2f8ad49edecc", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} })
56392025-03-07T22:38:54.502ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 538.384864ms iteration = 11 status = {}
56402025-03-07T22:38:54.507ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
56412025-03-07T22:38:54.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
56422025-03-07T22:38:54.508ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) }
56432025-03-07T22:38:54.509ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 545.196129ms iteration = 6 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
56442025-03-07T22:38:54.509ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
56452025-03-07T22:38:54.510ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "6417394c-b4bd-423b-9bdd-c39d9a65d2a3", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
56462025-03-07T22:38:54.510ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
56472025-03-07T22:38:54.512ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "1809dc55-ea43-4912-81fc-a983a8f3adad", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
56482025-03-07T22:38:54.512ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
56492025-03-07T22:38:54.515ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "27d20b63-fdf2-47f0-8763-475917b94331", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
56502025-03-07T22:38:54.515ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
56512025-03-07T22:38:54.524ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
56522025-03-07T22:38:54.524ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 560.423635ms iteration = 8 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
56532025-03-07T22:38:54.527ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
56542025-03-07T22:38:54.527ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
56552025-03-07T22:38:54.527ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
56562025-03-07T22:38:54.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
56572025-03-07T22:38:54.528ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
56582025-03-07T22:38:54.528ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
56592025-03-07T22:38:54.528ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
56602025-03-07T22:38:54.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
56612025-03-07T22:38:54.539ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56622025-03-07T22:38:54.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56632025-03-07T22:38:54.544ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56642025-03-07T22:38:54.549ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
56652025-03-07T22:38:54.549ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
56662025-03-07T22:38:54.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
56672025-03-07T22:38:54.567ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
56682025-03-07T22:38:54.568ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "4960870d-a9d5-48d4-b404-8aa344f37c88", "content-length": "691", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} })
56692025-03-07T22:38:54.568ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
56702025-03-07T22:38:54.568ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
56712025-03-07T22:38:54.568ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
56722025-03-07T22:38:54.570ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 606.244156ms iteration = 11 status = {}
56732025-03-07T22:38:54.577ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
56742025-03-07T22:38:54.577ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56752025-03-07T22:38:54.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56762025-03-07T22:38:54.581ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
56772025-03-07T22:38:54.581ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
56782025-03-07T22:38:54.583ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
56792025-03-07T22:38:54.587ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
56802025-03-07T22:38:54.591ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
56812025-03-07T22:38:54.591ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
56822025-03-07T22:38:54.591ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
56832025-03-07T22:38:54.591ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 627.849834ms iteration = 6 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
56842025-03-07T22:38:54.593ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
56852025-03-07T22:38:54.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
56862025-03-07T22:38:54.595ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
56872025-03-07T22:38:54.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
56882025-03-07T22:38:54.598ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
56892025-03-07T22:38:54.599ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "24918cc6-8032-49bc-aef2-cded1fb93a46", "content-length": "134", "date": "Fri, 07 Mar 2025 22:38:53 GMT"} })
56902025-03-07T22:38:54.599ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
56912025-03-07T22:38:54.599ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
56922025-03-07T22:38:54.599ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
56932025-03-07T22:38:54.599ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
56942025-03-07T22:38:54.599ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
56952025-03-07T22:38:54.599ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
56962025-03-07T22:38:54.599ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
56972025-03-07T22:38:54.599ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
56982025-03-07T22:38:54.599ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
56992025-03-07T22:38:54.599ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
57002025-03-07T22:38:54.600ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
57012025-03-07T22:38:54.604ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57022025-03-07T22:38:54.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57032025-03-07T22:38:54.606ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
57042025-03-07T22:38:54.606ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "edc86711-151d-48db-b854-4cbbffc8f12d", "content-length": "2", "date": "Fri, 07 Mar 2025 22:38:53 GMT"} })
57052025-03-07T22:38:54.607ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
57062025-03-07T22:38:54.607ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "78cfa3f5-926e-4582-9698-d60ffbb2f167", "content-length": "52", "date": "Fri, 07 Mar 2025 22:38:53 GMT"} })
57072025-03-07T22:38:54.607ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
57082025-03-07T22:38:54.607ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
57092025-03-07T22:38:54.624ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57102025-03-07T22:38:54.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57112025-03-07T22:38:54.627ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
57122025-03-07T22:38:54.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
57132025-03-07T22:38:54.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
57142025-03-07T22:38:54.628ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
57152025-03-07T22:38:54.628ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
57162025-03-07T22:38:54.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
57172025-03-07T22:38:54.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
57182025-03-07T22:38:54.630ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57192025-03-07T22:38:54.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
57202025-03-07T22:38:54.631ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
57212025-03-07T22:38:54.631ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
57222025-03-07T22:38:54.631ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-07 22:34:17.797135 UTC dns_group = external generation = 2
57232025-03-07T22:38:54.631ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 667.688536ms iteration = 6 status = {"generation":2}
57242025-03-07T22:38:54.633ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
57252025-03-07T22:38:54.633ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
57262025-03-07T22:38:54.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
57272025-03-07T22:38:54.635ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
57282025-03-07T22:38:54.635ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
57292025-03-07T22:38:54.635ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-07 22:34:17.208516 UTC dns_group = internal generation = 1
57302025-03-07T22:38:54.635ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 671.868677ms iteration = 6 status = {"generation":1}
57312025-03-07T22:38:54.654ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57322025-03-07T22:38:54.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57332025-03-07T22:38:54.660ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57342025-03-07T22:38:54.661ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
57352025-03-07T22:38:54.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
57362025-03-07T22:38:54.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
57372025-03-07T22:38:54.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
57382025-03-07T22:38:54.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
57392025-03-07T22:38:54.673ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 282 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 527e35a5-e530-495f-9c3d-0792c2b70d6b response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
57402025-03-07T22:38:54.689ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57412025-03-07T22:38:54.695ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
57422025-03-07T22:38:54.695ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57432025-03-07T22:38:54.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57442025-03-07T22:38:54.697ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57452025-03-07T22:38:54.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57462025-03-07T22:38:54.700ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57472025-03-07T22:38:54.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57482025-03-07T22:38:54.702ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57492025-03-07T22:38:54.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57502025-03-07T22:38:54.705ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57512025-03-07T22:38:54.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57522025-03-07T22:38:54.706ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57532025-03-07T22:38:54.707ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57542025-03-07T22:38:54.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57552025-03-07T22:38:54.708ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
57562025-03-07T22:38:54.708ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
57572025-03-07T22:38:54.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
57582025-03-07T22:38:54.711ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57592025-03-07T22:38:54.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57602025-03-07T22:38:54.712ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
57612025-03-07T22:38:54.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
57622025-03-07T22:38:54.713ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57632025-03-07T22:38:54.714ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57642025-03-07T22:38:54.715ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
57652025-03-07T22:38:54.715ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
57662025-03-07T22:38:54.716ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
57672025-03-07T22:38:54.718ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
57682025-03-07T22:38:54.718ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
57692025-03-07T22:38:54.718ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57702025-03-07T22:38:54.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
57712025-03-07T22:38:54.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57722025-03-07T22:38:54.720ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57732025-03-07T22:38:54.720ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57742025-03-07T22:38:54.720ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57752025-03-07T22:38:54.721ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57762025-03-07T22:38:54.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57772025-03-07T22:38:54.722ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
57782025-03-07T22:38:54.722ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
57792025-03-07T22:38:54.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
57802025-03-07T22:38:54.723ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57812025-03-07T22:38:54.723ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57822025-03-07T22:38:54.726ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
57832025-03-07T22:38:54.726ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
57842025-03-07T22:38:54.726ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
57852025-03-07T22:38:54.727ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
57862025-03-07T22:38:54.727ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57872025-03-07T22:38:54.727ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57882025-03-07T22:38:54.729ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
57892025-03-07T22:38:54.729ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 765.364675ms iteration = 3 status = {"blueprint_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","inventory_collection_id":"24b34bb5-d123-4c98-bdb5-62167bdbc6c1","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
57902025-03-07T22:38:54.731ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57912025-03-07T22:38:54.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57922025-03-07T22:38:54.733ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57932025-03-07T22:38:54.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57942025-03-07T22:38:54.735ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57952025-03-07T22:38:54.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57962025-03-07T22:38:54.737ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57972025-03-07T22:38:54.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57982025-03-07T22:38:54.739ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57992025-03-07T22:38:54.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58002025-03-07T22:38:54.741ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58012025-03-07T22:38:54.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58022025-03-07T22:38:54.742ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58032025-03-07T22:38:54.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58042025-03-07T22:38:54.743ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58052025-03-07T22:38:54.747ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
58062025-03-07T22:38:54.747ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58072025-03-07T22:38:54.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58082025-03-07T22:38:54.748ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58092025-03-07T22:38:54.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58102025-03-07T22:38:54.750ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58112025-03-07T22:38:54.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58122025-03-07T22:38:54.751ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58132025-03-07T22:38:54.755ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
58142025-03-07T22:38:54.755ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58152025-03-07T22:38:54.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58162025-03-07T22:38:54.761ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 00c422b8-3a81-4d2c-bc3f-fc426f6f9897, kind: Service { id: fbf58a59-609b-4fb3-b792-2c09f3f7b776 }, name: Name("external-dns-fbf58a59-609b-4fb3-b792-2c09f3f7b776"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 149, 171])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 066589e5-6cdf-4b01-9f39-98f2c968e6c2, kind: Service { id: 732b0e55-5ccf-4068-b771-1299f872f618 }, name: Name("external-dns-732b0e55-5ccf-4068-b771-1299f872f618"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 867d8023-aff2-4521-891f-07d5f45be915, kind: Service { id: dfe169a3-7536-4b54-9593-8815d6e81dd4 }, name: Name("nexus-dfe169a3-7536-4b54-9593-8815d6e81dd4"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 239])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8980b304-809c-4233-b9df-dd6e12e7c2f3, kind: Service { id: c0f69897-7f92-43db-ad47-96c3c4721128 }, name: Name("nexus-c0f69897-7f92-43db-ad47-96c3c4721128"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 221, 186])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: cb82146b-41bd-465f-8803-0f44afcd8f6e, kind: Service { id: 0f5c512c-2fda-4cff-bbad-41e8a283d9ff }, name: Name("nexus-0f5c512c-2fda-4cff-bbad-41e8a283d9ff"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 140, 207])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
58172025-03-07T22:38:54.761ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58182025-03-07T22:38:54.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58192025-03-07T22:38:54.763ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58202025-03-07T22:38:54.764ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
58212025-03-07T22:38:54.764ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58222025-03-07T22:38:54.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58232025-03-07T22:38:54.766ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
58242025-03-07T22:38:54.766ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 00c422b8-3a81-4d2c-bc3f-fc426f6f9897, kind: Service { id: fbf58a59-609b-4fb3-b792-2c09f3f7b776 }, name: Name("external-dns-fbf58a59-609b-4fb3-b792-2c09f3f7b776"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 149, 171])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 066589e5-6cdf-4b01-9f39-98f2c968e6c2, kind: Service { id: 732b0e55-5ccf-4068-b771-1299f872f618 }, name: Name("external-dns-732b0e55-5ccf-4068-b771-1299f872f618"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 867d8023-aff2-4521-891f-07d5f45be915, kind: Service { id: dfe169a3-7536-4b54-9593-8815d6e81dd4 }, name: Name("nexus-dfe169a3-7536-4b54-9593-8815d6e81dd4"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 239])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8980b304-809c-4233-b9df-dd6e12e7c2f3, kind: Service { id: c0f69897-7f92-43db-ad47-96c3c4721128 }, name: Name("nexus-c0f69897-7f92-43db-ad47-96c3c4721128"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 221, 186])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: cb82146b-41bd-465f-8803-0f44afcd8f6e, kind: Service { id: 0f5c512c-2fda-4cff-bbad-41e8a283d9ff }, name: Name("nexus-0f5c512c-2fda-4cff-bbad-41e8a283d9ff"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 140, 207])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
58252025-03-07T22:38:54.766ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
58262025-03-07T22:38:54.815ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "607981a2-46a8-467a-bd85-6d924b26fc9b", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} })
58272025-03-07T22:38:54.815ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
58282025-03-07T22:38:54.815ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
58292025-03-07T22:38:54.816ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f518c929-86a3-4d88-b4e2-069e43029f20", "content-length": "682", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} })
58302025-03-07T22:38:54.816ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58312025-03-07T22:38:54.816ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58322025-03-07T22:38:54.820ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58332025-03-07T22:38:54.820ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58342025-03-07T22:38:54.823ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58352025-03-07T22:38:54.823ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58362025-03-07T22:38:54.826ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58372025-03-07T22:38:54.826ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58382025-03-07T22:38:54.833ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 869.707218ms iteration = 11 status = {}
58392025-03-07T22:38:54.881ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, time_created: 2025-03-07T22:34:17.183069Z, time_modified: 2025-03-07T22:34:17.183069Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, is_scrimlet: true, serial_number: "buskin", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
58402025-03-07T22:38:54.881ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
58412025-03-07T22:38:54.883ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58422025-03-07T22:38:54.883ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58432025-03-07T22:38:54.885ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58442025-03-07T22:38:54.886ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58452025-03-07T22:38:54.887ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58462025-03-07T22:38:54.889ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
58472025-03-07T22:38:54.919ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
58482025-03-07T22:38:54.920ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "23f4d86f-4fff-43b1-96ea-85f3f9129cca", "date": "Fri, 07 Mar 2025 22:38:54 GMT"} })
58492025-03-07T22:38:54.920ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
58502025-03-07T22:38:54.920ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 957.258729ms
58512025-03-07T22:38:54.921ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 957.270629ms iteration = 2 status = {}
58522025-03-07T22:39:03.963ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
58532025-03-07T22:39:03.963ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 32 reason = Timeout
58542025-03-07T22:39:03.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58552025-03-07T22:39:03.963ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58562025-03-07T22:39:03.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58572025-03-07T22:39:03.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58582025-03-07T22:39:03.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58592025-03-07T22:39:03.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
58602025-03-07T22:39:03.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58612025-03-07T22:39:03.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58622025-03-07T22:39:03.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58632025-03-07T22:39:03.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58642025-03-07T22:39:03.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58652025-03-07T22:39:03.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
58662025-03-07T22:39:03.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58672025-03-07T22:39:03.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58682025-03-07T22:39:03.988ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
58692025-03-07T22:39:03.988ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.571307ms iteration = 32 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
58702025-03-07T22:39:03.995ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58712025-03-07T22:39:03.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58722025-03-07T22:39:04.031ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 67.540258ms iteration = 32 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
58732025-03-07T22:39:04.673ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 267 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 37629130-99fe-4d29-a37e-c48f8be03bcd response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
58742025-03-07T22:39:13.964ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
58752025-03-07T22:39:13.964ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 33 reason = Timeout
58762025-03-07T22:39:13.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58772025-03-07T22:39:13.964ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58782025-03-07T22:39:13.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58792025-03-07T22:39:13.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58802025-03-07T22:39:13.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58812025-03-07T22:39:13.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
58822025-03-07T22:39:13.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58832025-03-07T22:39:13.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58842025-03-07T22:39:13.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58852025-03-07T22:39:13.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58862025-03-07T22:39:13.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58872025-03-07T22:39:13.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
58882025-03-07T22:39:13.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58892025-03-07T22:39:13.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58902025-03-07T22:39:13.986ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
58912025-03-07T22:39:13.986ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.984811ms iteration = 33 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
58922025-03-07T22:39:13.995ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58932025-03-07T22:39:13.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58942025-03-07T22:39:14.023ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.116868ms iteration = 33 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
58952025-03-07T22:39:14.674ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 297 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 0f87c7e0-d69b-48d0-aa2b-506fabf9958a response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
58962025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 34 reason = Timeout
58972025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58982025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
58992025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
59002025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59012025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59022025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 12 reason = Timeout
59032025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 12 reason = Timeout
59042025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
59052025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59062025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 12 reason = Timeout
59072025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
59082025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 12 reason = Timeout
59092025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 12 reason = Timeout
59102025-03-07T22:39:23.964ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
59112025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59122025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
59132025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 12 reason = Timeout
59142025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
59152025-03-07T22:39:23.964ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
59162025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 12 reason = Timeout
59172025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
59182025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 13 reason = Timeout
59192025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
59202025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
59212025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
59222025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
59232025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
59242025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
59252025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
59262025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
59272025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 12 reason = Timeout
59282025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
59292025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 12 reason = Timeout
59302025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
59312025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 12 reason = Timeout
59322025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 12 reason = Timeout
59332025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
59342025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
59352025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59362025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59372025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
59382025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
59392025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59402025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
59412025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59422025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
59432025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
59442025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
59452025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
59462025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
59472025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
59482025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
59492025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
59502025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
59512025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
59522025-03-07T22:39:23.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
59532025-03-07T22:39:23.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59542025-03-07T22:39:23.968ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59552025-03-07T22:39:23.968ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.655605ms iteration = 12 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
59562025-03-07T22:39:23.968ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59572025-03-07T22:39:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
59582025-03-07T22:39:23.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59592025-03-07T22:39:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59602025-03-07T22:39:23.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
59612025-03-07T22:39:23.971ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59622025-03-07T22:39:23.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59632025-03-07T22:39:23.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
59642025-03-07T22:39:23.971ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59652025-03-07T22:39:23.971ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59662025-03-07T22:39:23.971ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59672025-03-07T22:39:23.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
59682025-03-07T22:39:23.973ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59692025-03-07T22:39:23.973ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59702025-03-07T22:39:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
59712025-03-07T22:39:23.973ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
59722025-03-07T22:39:23.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
59732025-03-07T22:39:23.974ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
59742025-03-07T22:39:23.974ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59752025-03-07T22:39:23.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59762025-03-07T22:39:23.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
59772025-03-07T22:39:23.975ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
59782025-03-07T22:39:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
59792025-03-07T22:39:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
59802025-03-07T22:39:23.976ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
59812025-03-07T22:39:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
59822025-03-07T22:39:23.976ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1122, tv_nsec: 330335813 } })
59832025-03-07T22:39:23.977ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
59842025-03-07T22:39:23.978ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1122, tv_nsec: 332361863 } })
59852025-03-07T22:39:23.979ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
59862025-03-07T22:39:23.980ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59872025-03-07T22:39:23.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59882025-03-07T22:39:23.982ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59892025-03-07T22:39:23.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
59902025-03-07T22:39:23.983ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
59912025-03-07T22:39:23.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59922025-03-07T22:39:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59932025-03-07T22:39:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
59942025-03-07T22:39:23.987ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1122, tv_nsec: 340598702 } })
59952025-03-07T22:39:23.987ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
59962025-03-07T22:39:23.990ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59972025-03-07T22:39:23.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
59982025-03-07T22:39:24.007ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59992025-03-07T22:39:24.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
60002025-03-07T22:39:24.009ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 44.74247ms iteration = 34 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
60012025-03-07T22:39:24.013ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
60022025-03-07T22:39:24.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
60032025-03-07T22:39:24.020ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
60042025-03-07T22:39:24.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60052025-03-07T22:39:24.021ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
60062025-03-07T22:39:24.021ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
60072025-03-07T22:39:24.021ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
60082025-03-07T22:39:24.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
60092025-03-07T22:39:24.024ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
60102025-03-07T22:39:24.024ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 59.869035ms iteration = 12 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
60112025-03-07T22:39:24.025ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60122025-03-07T22:39:24.026ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 61.40399ms iteration = 12 status = {"errors":[],"garbage_collect_requested":[]}
60132025-03-07T22:39:24.028ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
60142025-03-07T22:39:24.028ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
60152025-03-07T22:39:24.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60162025-03-07T22:39:24.029ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60172025-03-07T22:39:24.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60182025-03-07T22:39:24.029ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
60192025-03-07T22:39:24.029ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 64.953323ms iteration = 12 status = {"errors":[],"finish_invoked_ok":[]}
60202025-03-07T22:39:24.030ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60212025-03-07T22:39:24.030ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60222025-03-07T22:39:24.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
60232025-03-07T22:39:24.031ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
60242025-03-07T22:39:24.031ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 66.315217ms iteration = 12 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
60252025-03-07T22:39:24.032ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
60262025-03-07T22:39:24.032ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60272025-03-07T22:39:24.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60282025-03-07T22:39:24.032ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
60292025-03-07T22:39:24.032ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60302025-03-07T22:39:24.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60312025-03-07T22:39:24.035ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 70.188754ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
60322025-03-07T22:39:24.035ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60332025-03-07T22:39:24.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60342025-03-07T22:39:24.037ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60352025-03-07T22:39:24.040ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
60362025-03-07T22:39:24.040ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60372025-03-07T22:39:24.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60382025-03-07T22:39:24.042ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
60392025-03-07T22:39:24.042ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 77.877588ms iteration = 12 status = {"errors":[],"requests_created_ok":[]}
60402025-03-07T22:39:24.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
60412025-03-07T22:39:24.043ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 78.225822ms iteration = 12 status = 0
60422025-03-07T22:39:24.054ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
60432025-03-07T22:39:24.054ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 89.50808ms iteration = 34 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
60442025-03-07T22:39:24.068ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
60452025-03-07T22:39:24.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
60462025-03-07T22:39:24.072ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
60472025-03-07T22:39:24.072ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
60482025-03-07T22:39:24.073ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd9160b1-1085-48d4-bd55-544bce962236", "content-length": "26", "date": "Fri, 07 Mar 2025 22:39:24 GMT"} })
60492025-03-07T22:39:24.073ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
60502025-03-07T22:39:24.075ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
60512025-03-07T22:39:24.078ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
60522025-03-07T22:39:24.078ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
60532025-03-07T22:39:24.078ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
60542025-03-07T22:39:24.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
60552025-03-07T22:39:24.099ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 134.126529ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
60562025-03-07T22:39:24.126ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 161.695613ms iteration = 12 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
60572025-03-07T22:39:24.129ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
60582025-03-07T22:39:24.129ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
60592025-03-07T22:39:24.129ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 164.905165ms iteration = 12 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
60602025-03-07T22:39:24.141ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
60612025-03-07T22:39:24.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
60622025-03-07T22:39:24.171ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 206.08318ms iteration = 13 status = {"physical_disks_added":0}
60632025-03-07T22:39:24.171ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
60642025-03-07T22:39:24.171ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
60652025-03-07T22:39:24.172ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bdfffd0-7915-4829-9437-3785f950a3e3", "content-length": "26", "date": "Fri, 07 Mar 2025 22:39:24 GMT"} })
60662025-03-07T22:39:24.172ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
60672025-03-07T22:39:24.174ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
60682025-03-07T22:39:24.199ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
60692025-03-07T22:39:24.199ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "9217d547-ae52-47e9-bac3-b93891f46c6d", "content-length": "1", "date": "Fri, 07 Mar 2025 22:39:24 GMT"} })
60702025-03-07T22:39:24.201ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
60712025-03-07T22:39:24.201ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
60722025-03-07T22:39:24.201ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2072ec9d-e6eb-405b-bb46-e5a2cc35b25c", "content-length": "26", "date": "Fri, 07 Mar 2025 22:39:24 GMT"} })
60732025-03-07T22:39:24.201ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
60742025-03-07T22:39:24.202ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
60752025-03-07T22:39:24.202ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
60762025-03-07T22:39:24.203ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
60772025-03-07T22:39:24.205ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 240.485491ms iteration = 12 status = null
60782025-03-07T22:39:24.221ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
60792025-03-07T22:39:24.221ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
60802025-03-07T22:39:24.221ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e507bcc-842e-4a1f-8ff3-658a23b96bb0", "content-length": "372", "date": "Fri, 07 Mar 2025 22:39:24 GMT"} })
60812025-03-07T22:39:24.221ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60822025-03-07T22:39:24.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60832025-03-07T22:39:24.239ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
60842025-03-07T22:39:24.239ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
60852025-03-07T22:39:24.239ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 274.456927ms iteration = 12 status = {}
60862025-03-07T22:39:24.240ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60872025-03-07T22:39:24.242ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
60882025-03-07T22:39:24.242ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
60892025-03-07T22:39:24.242ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
60902025-03-07T22:39:24.244ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60912025-03-07T22:39:24.245ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
60922025-03-07T22:39:24.245ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
60932025-03-07T22:39:24.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
60942025-03-07T22:39:24.269ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
60952025-03-07T22:39:24.269ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
60962025-03-07T22:39:24.270ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "e5de3d82-c5e1-4683-a12f-aa59a90af2e6", "date": "Fri, 07 Mar 2025 22:39:24 GMT"} })
60972025-03-07T22:39:24.270ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60982025-03-07T22:39:24.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60992025-03-07T22:39:24.272ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61002025-03-07T22:39:24.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61012025-03-07T22:39:24.275ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61022025-03-07T22:39:24.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61032025-03-07T22:39:24.279ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
61042025-03-07T22:39:24.279ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
61052025-03-07T22:39:24.279ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "771c90e0-25a2-4f0c-ad2e-e85ec7b65aae", "date": "Fri, 07 Mar 2025 22:39:24 GMT"} })
61062025-03-07T22:39:24.280ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 315.854535ms iteration = 12 status = {}
61072025-03-07T22:39:24.319ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
61082025-03-07T22:39:24.320ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "22aab7b8-4f53-47ed-9afa-56509e187911", "content-length": "691", "date": "Fri, 07 Mar 2025 22:39:24 GMT"} })
61092025-03-07T22:39:24.320ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
61102025-03-07T22:39:24.320ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
61112025-03-07T22:39:24.320ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
61122025-03-07T22:39:24.322ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 357.217822ms iteration = 12 status = {}
61132025-03-07T22:39:24.340ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61142025-03-07T22:39:24.341ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61152025-03-07T22:39:24.342ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
61162025-03-07T22:39:24.342ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61172025-03-07T22:39:24.342ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61182025-03-07T22:39:24.352ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
61192025-03-07T22:39:24.352ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bce6583-9eea-44dc-8d00-394145c935a9", "content-length": "134", "date": "Fri, 07 Mar 2025 22:39:24 GMT"} })
61202025-03-07T22:39:24.352ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
61212025-03-07T22:39:24.352ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
61222025-03-07T22:39:24.352ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
61232025-03-07T22:39:24.352ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
61242025-03-07T22:39:24.353ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
61252025-03-07T22:39:24.353ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
61262025-03-07T22:39:24.353ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
61272025-03-07T22:39:24.353ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
61282025-03-07T22:39:24.353ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
61292025-03-07T22:39:24.353ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
61302025-03-07T22:39:24.353ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
61312025-03-07T22:39:24.353ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61322025-03-07T22:39:24.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61332025-03-07T22:39:24.355ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
61342025-03-07T22:39:24.355ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "430b319c-dd83-48eb-a8fd-57b53a7676ca", "content-length": "2", "date": "Fri, 07 Mar 2025 22:39:24 GMT"} })
61352025-03-07T22:39:24.355ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
61362025-03-07T22:39:24.355ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "d28f7cd6-aa35-4da4-abe5-f5c60a01af46", "content-length": "52", "date": "Fri, 07 Mar 2025 22:39:24 GMT"} })
61372025-03-07T22:39:24.355ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
61382025-03-07T22:39:24.355ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
61392025-03-07T22:39:24.482ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "84e266d7-597f-4b57-b48e-49d4faf6b54d", "date": "Fri, 07 Mar 2025 22:39:24 GMT"} })
61402025-03-07T22:39:24.482ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
61412025-03-07T22:39:24.482ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
61422025-03-07T22:39:24.483ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "094b98b2-9eb6-4c7e-89e8-b197b64f2f44", "content-length": "682", "date": "Fri, 07 Mar 2025 22:39:24 GMT"} })
61432025-03-07T22:39:24.483ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61442025-03-07T22:39:24.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61452025-03-07T22:39:24.487ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61462025-03-07T22:39:24.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61472025-03-07T22:39:24.492ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61482025-03-07T22:39:24.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61492025-03-07T22:39:24.495ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61502025-03-07T22:39:24.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61512025-03-07T22:39:24.503ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 538.713477ms iteration = 12 status = {}
61522025-03-07T22:39:24.674ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 262 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 3c3205ac-8c37-43d7-9ae2-4ed2b94a12fc response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
61532025-03-07T22:39:33.964ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
61542025-03-07T22:39:33.965ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 35 reason = Timeout
61552025-03-07T22:39:33.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61562025-03-07T22:39:33.965ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
61572025-03-07T22:39:33.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61582025-03-07T22:39:33.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
61592025-03-07T22:39:33.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61602025-03-07T22:39:33.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
61612025-03-07T22:39:33.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61622025-03-07T22:39:33.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61632025-03-07T22:39:33.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61642025-03-07T22:39:33.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61652025-03-07T22:39:33.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61662025-03-07T22:39:33.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
61672025-03-07T22:39:33.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61682025-03-07T22:39:33.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61692025-03-07T22:39:33.988ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
61702025-03-07T22:39:33.988ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.445016ms iteration = 35 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
61712025-03-07T22:39:33.996ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
61722025-03-07T22:39:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
61732025-03-07T22:39:33.998ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.045317ms iteration = 35 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
61742025-03-07T22:39:34.674ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 280 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = d4e23c5d-29d6-4645-83d1-f0439c087d9d response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
61752025-03-07T22:39:43.965ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
61762025-03-07T22:39:43.965ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 36 reason = Timeout
61772025-03-07T22:39:43.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61782025-03-07T22:39:43.965ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
61792025-03-07T22:39:43.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61802025-03-07T22:39:43.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
61812025-03-07T22:39:43.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61822025-03-07T22:39:43.967ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
61832025-03-07T22:39:43.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
61842025-03-07T22:39:43.969ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.10377ms iteration = 36 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
61852025-03-07T22:39:43.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
61862025-03-07T22:39:43.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61872025-03-07T22:39:43.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61882025-03-07T22:39:43.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61892025-03-07T22:39:43.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61902025-03-07T22:39:43.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61912025-03-07T22:39:43.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
61922025-03-07T22:39:43.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61932025-03-07T22:39:43.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61942025-03-07T22:39:43.983ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
61952025-03-07T22:39:43.984ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.993693ms iteration = 36 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
61962025-03-07T22:39:44.674ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 265 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 34f219ea-cbb7-4879-9309-0f049265ca27 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
61972025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 14 reason = Timeout
61982025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
61992025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
62002025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
62012025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
62022025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
62032025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 7 reason = Timeout
62042025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
62052025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
62062025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
62072025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 7 reason = Timeout
62082025-03-07T22:39:53.965ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:29:53.965225797 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
62092025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 4 reason = Timeout
62102025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
62112025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
62122025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
62132025-03-07T22:39:53.965ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
62142025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 13 reason = Timeout
62152025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
62162025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.41\u{b5}s iteration = 8 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
62172025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 13 reason = Timeout
62182025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
62192025-03-07T22:39:53.965ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
62202025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 13 reason = Timeout
62212025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
62222025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
62232025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
62242025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 13 reason = Timeout
62252025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 13 reason = Timeout
62262025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
62272025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62282025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
62292025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
62302025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 37 reason = Timeout
62312025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62322025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 7 reason = Timeout
62332025-03-07T22:39:53.965ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
62342025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
62352025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
62362025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
62372025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
62382025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
62392025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
62402025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
62412025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
62422025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
62432025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
62442025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
62452025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
62462025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
62472025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
62482025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 13 reason = Timeout
62492025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62502025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
62512025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 13 reason = Timeout
62522025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
62532025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 13 reason = Timeout
62542025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
62552025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 7 reason = Timeout
62562025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
62572025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
62582025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
62592025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 7 reason = Timeout
62602025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
62612025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
62622025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 13 reason = Timeout
62632025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 13 reason = Timeout
62642025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
62652025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 13 reason = Timeout
62662025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
62672025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
62682025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
62692025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
62702025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
62712025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
62722025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62732025-03-07T22:39:53.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
62742025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
62752025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
62762025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
62772025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
62782025-03-07T22:39:53.966ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1152, tv_nsec: 319059505 } })
62792025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
62802025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
62812025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62822025-03-07T22:39:53.966ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1152, tv_nsec: 319094665 } })
62832025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62842025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
62852025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
62862025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
62872025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
62882025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62892025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
62902025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
62912025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
62922025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
62932025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
62942025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
62952025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
62962025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
62972025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
62982025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
62992025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
63002025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.422884ms iteration = 7 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
63012025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
63022025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
63032025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
63042025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.193222ms iteration = 7 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
63052025-03-07T22:39:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
63062025-03-07T22:39:53.967ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63072025-03-07T22:39:53.968ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.388663ms iteration = 13 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
63082025-03-07T22:39:53.969ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63092025-03-07T22:39:53.969ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63102025-03-07T22:39:53.969ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1152, tv_nsec: 322290306 } })
63112025-03-07T22:39:53.969ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
63122025-03-07T22:39:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
63132025-03-07T22:39:53.969ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
63142025-03-07T22:39:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
63152025-03-07T22:39:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
63162025-03-07T22:39:53.971ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
63172025-03-07T22:39:53.971ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
63182025-03-07T22:39:53.971ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63192025-03-07T22:39:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
63202025-03-07T22:39:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63212025-03-07T22:39:53.972ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63222025-03-07T22:39:53.972ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:29:53.965225797 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
63232025-03-07T22:39:53.972ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 7.26116ms iteration = 7 status = {"expiration":"2025-03-07T22:29:53.965225797Z","pruned":[]}
63242025-03-07T22:39:53.972ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
63252025-03-07T22:39:53.973ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63262025-03-07T22:39:53.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
63272025-03-07T22:39:53.973ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63282025-03-07T22:39:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
63292025-03-07T22:39:53.974ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63302025-03-07T22:39:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63312025-03-07T22:39:53.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
63322025-03-07T22:39:53.975ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
63332025-03-07T22:39:53.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
63342025-03-07T22:39:53.977ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
63352025-03-07T22:39:53.977ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
63362025-03-07T22:39:53.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
63372025-03-07T22:39:53.981ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63382025-03-07T22:39:53.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63392025-03-07T22:39:53.981ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
63402025-03-07T22:39:53.981ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
63412025-03-07T22:39:53.981ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 16.062634ms iteration = 7 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
63422025-03-07T22:39:53.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63432025-03-07T22:39:53.982ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63442025-03-07T22:39:53.982ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63452025-03-07T22:39:53.984ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
63462025-03-07T22:39:53.984ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
63472025-03-07T22:39:53.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
63482025-03-07T22:39:53.985ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
63492025-03-07T22:39:53.985ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
63502025-03-07T22:39:53.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
63512025-03-07T22:39:53.986ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
63522025-03-07T22:39:53.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63532025-03-07T22:39:53.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63542025-03-07T22:39:53.986ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
63552025-03-07T22:39:53.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
63562025-03-07T22:39:53.988ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63572025-03-07T22:39:53.990ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1152, tv_nsec: 343040015 } })
63582025-03-07T22:39:53.990ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 24.618617ms iteration = 7 status = {"errors":[],"found_port_ok":[]}
63592025-03-07T22:39:53.990ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
63602025-03-07T22:39:53.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
63612025-03-07T22:39:53.991ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
63622025-03-07T22:39:53.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
63632025-03-07T22:39:53.991ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
63642025-03-07T22:39:53.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
63652025-03-07T22:39:53.993ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63662025-03-07T22:39:53.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
63672025-03-07T22:39:53.995ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
63682025-03-07T22:39:53.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
63692025-03-07T22:39:53.996ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63702025-03-07T22:39:54.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
63712025-03-07T22:39:54.002ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
63722025-03-07T22:39:54.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
63732025-03-07T22:39:54.002ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
63742025-03-07T22:39:54.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
63752025-03-07T22:39:54.003ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63762025-03-07T22:39:54.005ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63772025-03-07T22:39:54.006ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
63782025-03-07T22:39:54.006ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
63792025-03-07T22:39:54.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
63802025-03-07T22:39:54.007ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
63812025-03-07T22:39:54.007ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63822025-03-07T22:39:54.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63832025-03-07T22:39:54.009ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63842025-03-07T22:39:54.010ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1152, tv_nsec: 363295330 } })
63852025-03-07T22:39:54.010ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
63862025-03-07T22:39:54.010ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
63872025-03-07T22:39:54.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
63882025-03-07T22:39:54.012ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
63892025-03-07T22:39:54.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
63902025-03-07T22:39:54.013ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
63912025-03-07T22:39:54.014ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
63922025-03-07T22:39:54.014ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 48.787489ms iteration = 13 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
63932025-03-07T22:39:54.015ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
63942025-03-07T22:39:54.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
63952025-03-07T22:39:54.015ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63962025-03-07T22:39:54.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63972025-03-07T22:39:54.016ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63982025-03-07T22:39:54.016ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
63992025-03-07T22:39:54.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
64002025-03-07T22:39:54.019ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 53.757077ms iteration = 13 status = {"errors":[],"requests_created_ok":[]}
64012025-03-07T22:39:54.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
64022025-03-07T22:39:54.021ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 56.010348ms iteration = 13 status = {"errors":[],"finish_invoked_ok":[]}
64032025-03-07T22:39:54.024ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
64042025-03-07T22:39:54.024ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64052025-03-07T22:39:54.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
64062025-03-07T22:39:54.025ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64072025-03-07T22:39:54.026ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64082025-03-07T22:39:54.026ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
64092025-03-07T22:39:54.026ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64102025-03-07T22:39:54.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64112025-03-07T22:39:54.027ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
64122025-03-07T22:39:54.027ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
64132025-03-07T22:39:54.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
64142025-03-07T22:39:54.028ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
64152025-03-07T22:39:54.028ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64162025-03-07T22:39:54.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64172025-03-07T22:39:54.040ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
64182025-03-07T22:39:54.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
64192025-03-07T22:39:54.054ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64202025-03-07T22:39:54.056ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
64212025-03-07T22:39:54.056ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
64222025-03-07T22:39:54.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
64232025-03-07T22:39:54.057ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 91.791201ms iteration = 13 status = {"errors":[],"garbage_collect_requested":[]}
64242025-03-07T22:39:54.060ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
64252025-03-07T22:39:54.060ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
64262025-03-07T22:39:54.060ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
64272025-03-07T22:39:54.060ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
64282025-03-07T22:39:54.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
64292025-03-07T22:39:54.061ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64302025-03-07T22:39:54.061ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
64312025-03-07T22:39:54.061ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64322025-03-07T22:39:54.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64332025-03-07T22:39:54.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64342025-03-07T22:39:54.066ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
64352025-03-07T22:39:54.066ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
64362025-03-07T22:39:54.066ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 101.291463ms iteration = 7 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
64372025-03-07T22:39:54.066ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 101.642836ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
64382025-03-07T22:39:54.071ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
64392025-03-07T22:39:54.076ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 111.224568ms iteration = 37 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
64402025-03-07T22:39:54.077ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64412025-03-07T22:39:54.077ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
64422025-03-07T22:39:54.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
64432025-03-07T22:39:54.080ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
64442025-03-07T22:39:54.080ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64452025-03-07T22:39:54.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64462025-03-07T22:39:54.080ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
64472025-03-07T22:39:54.080ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
64482025-03-07T22:39:54.080ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
64492025-03-07T22:39:54.080ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
64502025-03-07T22:39:54.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
64512025-03-07T22:39:54.095ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
64522025-03-07T22:39:54.095ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
64532025-03-07T22:39:54.096ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d3e9e82-20c3-44df-a226-573b48d829a7", "content-length": "26", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} })
64542025-03-07T22:39:54.096ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
64552025-03-07T22:39:54.098ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
64562025-03-07T22:39:54.112ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
64572025-03-07T22:39:54.112ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
64582025-03-07T22:39:54.113ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e11419d-19fb-42ab-9f43-ba1f8d5724b0", "content-length": "26", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} })
64592025-03-07T22:39:54.113ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
64602025-03-07T22:39:54.114ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
64612025-03-07T22:39:54.156ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
64622025-03-07T22:39:54.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
64632025-03-07T22:39:54.196ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
64642025-03-07T22:39:54.196ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
64652025-03-07T22:39:54.197ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ff7a6ee-4d42-4af5-874d-0357b7257aaa", "content-length": "372", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} })
64662025-03-07T22:39:54.197ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
64672025-03-07T22:39:54.197ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
64682025-03-07T22:39:54.199ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
64692025-03-07T22:39:54.199ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
64702025-03-07T22:39:54.199ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb3c922e-64f3-42c7-93d7-27d1ef54846e", "content-length": "26", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} })
64712025-03-07T22:39:54.199ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
64722025-03-07T22:39:54.200ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 234.707335ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
64732025-03-07T22:39:54.200ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64742025-03-07T22:39:54.200ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
64752025-03-07T22:39:54.200ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64762025-03-07T22:39:54.200ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64772025-03-07T22:39:54.200ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 235.552753ms iteration = 4 status = {"nsuccess":5}
64782025-03-07T22:39:54.200ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64792025-03-07T22:39:54.201ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
64802025-03-07T22:39:54.201ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64812025-03-07T22:39:54.203ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64822025-03-07T22:39:54.203ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64832025-03-07T22:39:54.204ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
64842025-03-07T22:39:54.204ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64852025-03-07T22:39:54.204ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64862025-03-07T22:39:54.205ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
64872025-03-07T22:39:54.205ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
64882025-03-07T22:39:54.205ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
64892025-03-07T22:39:54.207ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64902025-03-07T22:39:54.207ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64912025-03-07T22:39:54.209ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
64922025-03-07T22:39:54.209ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
64932025-03-07T22:39:54.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
64942025-03-07T22:39:54.211ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
64952025-03-07T22:39:54.211ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
64962025-03-07T22:39:54.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
64972025-03-07T22:39:54.212ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
64982025-03-07T22:39:54.212ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 246.815871ms iteration = 37 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
64992025-03-07T22:39:54.213ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65002025-03-07T22:39:54.216ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
65012025-03-07T22:39:54.216ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65022025-03-07T22:39:54.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65032025-03-07T22:39:54.227ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 262.31729ms iteration = 13 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
65042025-03-07T22:39:54.243ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65052025-03-07T22:39:54.243ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
65062025-03-07T22:39:54.245ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
65072025-03-07T22:39:54.245ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
65082025-03-07T22:39:54.245ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
65092025-03-07T22:39:54.245ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 280.178742ms iteration = 13 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
65102025-03-07T22:39:54.246ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
65112025-03-07T22:39:54.246ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65122025-03-07T22:39:54.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65132025-03-07T22:39:54.247ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
65142025-03-07T22:39:54.248ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "c8e43245-2712-44ed-a5e9-5fc9743fda0d", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
65152025-03-07T22:39:54.248ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
65162025-03-07T22:39:54.249ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 284.394793ms iteration = 13 status = 0
65172025-03-07T22:39:54.250ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "c8eb6a5f-a3af-463e-8398-e398713da77c", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
65182025-03-07T22:39:54.250ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
65192025-03-07T22:39:54.252ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
65202025-03-07T22:39:54.252ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 287.053048ms iteration = 9 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
65212025-03-07T22:39:54.255ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
65222025-03-07T22:39:54.255ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
65232025-03-07T22:39:54.255ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
65242025-03-07T22:39:54.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
65252025-03-07T22:39:54.257ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
65262025-03-07T22:39:54.257ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
65272025-03-07T22:39:54.257ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 291.923915ms iteration = 13 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
65282025-03-07T22:39:54.257ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65292025-03-07T22:39:54.264ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
65302025-03-07T22:39:54.264ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
65312025-03-07T22:39:54.264ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
65322025-03-07T22:39:54.268ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
65332025-03-07T22:39:54.268ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
65342025-03-07T22:39:54.268ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "96917f84-89cc-4419-8a4a-47d7790f5f19", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} })
65352025-03-07T22:39:54.268ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65362025-03-07T22:39:54.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65372025-03-07T22:39:54.269ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
65382025-03-07T22:39:54.269ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
65392025-03-07T22:39:54.269ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
65402025-03-07T22:39:54.269ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
65412025-03-07T22:39:54.269ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
65422025-03-07T22:39:54.269ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-07 22:34:17.208516 UTC dns_group = internal generation = 1
65432025-03-07T22:39:54.269ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 304.491126ms iteration = 7 status = {"generation":1}
65442025-03-07T22:39:54.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
65452025-03-07T22:39:54.271ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65462025-03-07T22:39:54.271ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65472025-03-07T22:39:54.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65482025-03-07T22:39:54.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
65492025-03-07T22:39:54.276ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
65502025-03-07T22:39:54.276ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65512025-03-07T22:39:54.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
65522025-03-07T22:39:54.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65532025-03-07T22:39:54.278ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
65542025-03-07T22:39:54.278ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
65552025-03-07T22:39:54.278ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-07 22:34:17.797135 UTC dns_group = external generation = 2
65562025-03-07T22:39:54.278ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 312.902357ms iteration = 7 status = {"generation":2}
65572025-03-07T22:39:54.281ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
65582025-03-07T22:39:54.281ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
65592025-03-07T22:39:54.282ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "04f90aa6-7f6d-4989-8462-c88841a6296a", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} })
65602025-03-07T22:39:54.284ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 318.948264ms iteration = 13 status = {}
65612025-03-07T22:39:54.304ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65622025-03-07T22:39:54.307ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
65632025-03-07T22:39:54.307ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
65642025-03-07T22:39:54.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
65652025-03-07T22:39:54.342ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
65662025-03-07T22:39:54.342ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 377.472987ms iteration = 7 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
65672025-03-07T22:39:54.346ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 381.230383ms iteration = 14 status = {"physical_disks_added":0}
65682025-03-07T22:39:54.348ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 382.98501ms iteration = 13 status = {}
65692025-03-07T22:39:54.361ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
65702025-03-07T22:39:54.361ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "445c5b7e-9b1a-40fd-9d93-e0520c3d9213", "content-length": "1", "date": "Fri, 07 Mar 2025 22:39:53 GMT"} })
65712025-03-07T22:39:54.363ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
65722025-03-07T22:39:54.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
65732025-03-07T22:39:54.365ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 399.782211ms iteration = 13 status = null
65742025-03-07T22:39:54.375ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
65752025-03-07T22:39:54.376ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "ef8c98af-f97e-4325-92e7-ccba3f911d33", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
65762025-03-07T22:39:54.376ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
65772025-03-07T22:39:54.377ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "e902d1b0-4130-446c-9435-264b6cd05fb9", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
65782025-03-07T22:39:54.377ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
65792025-03-07T22:39:54.379ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "4e3ae972-b836-4b64-ad27-1fe7b96317d6", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
65802025-03-07T22:39:54.379ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
65812025-03-07T22:39:54.382ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
65822025-03-07T22:39:54.382ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 416.988656ms iteration = 9 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
65832025-03-07T22:39:54.386ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
65842025-03-07T22:39:54.386ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fb078df-552d-48fc-af69-aea64437cf25", "content-length": "691", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} })
65852025-03-07T22:39:54.386ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
65862025-03-07T22:39:54.386ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
65872025-03-07T22:39:54.386ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
65882025-03-07T22:39:54.388ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 423.102016ms iteration = 13 status = {}
65892025-03-07T22:39:54.436ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65902025-03-07T22:39:54.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65912025-03-07T22:39:54.438ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
65922025-03-07T22:39:54.438ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65932025-03-07T22:39:54.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65942025-03-07T22:39:54.446ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
65952025-03-07T22:39:54.447ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cb6dc79-c0ff-4185-84a6-d62c7f914a04", "content-length": "134", "date": "Fri, 07 Mar 2025 22:39:53 GMT"} })
65962025-03-07T22:39:54.447ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
65972025-03-07T22:39:54.447ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
65982025-03-07T22:39:54.447ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
65992025-03-07T22:39:54.447ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
66002025-03-07T22:39:54.447ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66012025-03-07T22:39:54.447ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66022025-03-07T22:39:54.447ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66032025-03-07T22:39:54.447ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
66042025-03-07T22:39:54.447ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66052025-03-07T22:39:54.447ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
66062025-03-07T22:39:54.448ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
66072025-03-07T22:39:54.448ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66082025-03-07T22:39:54.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66092025-03-07T22:39:54.456ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
66102025-03-07T22:39:54.456ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd26c169-844a-4767-897c-b7e9e6092925", "content-length": "2", "date": "Fri, 07 Mar 2025 22:39:53 GMT"} })
66112025-03-07T22:39:54.456ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
66122025-03-07T22:39:54.456ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb71a98e-0cd5-47fc-b3a6-9a0c66aac571", "content-length": "52", "date": "Fri, 07 Mar 2025 22:39:53 GMT"} })
66132025-03-07T22:39:54.456ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
66142025-03-07T22:39:54.456ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
66152025-03-07T22:39:54.458ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) }
66162025-03-07T22:39:54.458ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 493.025218ms iteration = 7 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
66172025-03-07T22:39:54.587ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "d0f63dc5-de12-45f6-9c13-67386b3401cb", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} })
66182025-03-07T22:39:54.587ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
66192025-03-07T22:39:54.587ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
66202025-03-07T22:39:54.588ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d256025-48c5-4aab-a380-85aff8337deb", "content-length": "682", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} })
66212025-03-07T22:39:54.588ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66222025-03-07T22:39:54.588ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66232025-03-07T22:39:54.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66242025-03-07T22:39:54.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66252025-03-07T22:39:54.597ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66262025-03-07T22:39:54.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66272025-03-07T22:39:54.599ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66282025-03-07T22:39:54.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66292025-03-07T22:39:54.606ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 641.134071ms iteration = 13 status = {}
66302025-03-07T22:39:54.659ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
66312025-03-07T22:39:54.659ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
66322025-03-07T22:39:54.689ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
66332025-03-07T22:39:54.689ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:41885
66342025-03-07T22:39:54.689ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:41885 req_id = 5347d9b6-41ae-4bb2-9577-e4fe704e14bb roles = RoleSet { roles: {} } uri = /metrics/producers
66352025-03-07T22:39:54.689ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 245 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 57d5f404-3f95-45f6-ba58-cfe054010f70 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
66362025-03-07T22:39:54.689ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:41885 req_id = 5347d9b6-41ae-4bb2-9577-e4fe704e14bb resource = Database result = Ok(()) uri = /metrics/producers
66372025-03-07T22:39:54.732ZINFOnexus: assigned collector to new producer collector_id = 7b5e9a24-6b39-4a58-a275-8cc5a75195d8 file = nexus/src/app/oximeter.rs:76 producer_id = dfe169a3-7536-4b54-9593-8815d6e81dd4
66382025-03-07T22:39:54.732ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 42928 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:41885 req_id = 5347d9b6-41ae-4bb2-9577-e4fe704e14bb response_code = 201 uri = /metrics/producers
66392025-03-07T22:39:54.732ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "5347d9b6-41ae-4bb2-9577-e4fe704e14bb", "content-length": "41", "date": "Fri, 07 Mar 2025 22:39:54 GMT"} })
66402025-03-07T22:39:54.733ZDEBGnexus (producer-registration-task): registered with nexus successfully
66412025-03-07T22:39:54.733ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
66422025-03-07T22:40:03.965ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 38 reason = Timeout
66432025-03-07T22:40:03.965ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66442025-03-07T22:40:03.965ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
66452025-03-07T22:40:03.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66462025-03-07T22:40:03.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66472025-03-07T22:40:03.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66482025-03-07T22:40:03.995ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66492025-03-07T22:40:03.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66502025-03-07T22:40:03.996ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66512025-03-07T22:40:03.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
66522025-03-07T22:40:03.998ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66532025-03-07T22:40:03.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66542025-03-07T22:40:04.001ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66552025-03-07T22:40:04.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66562025-03-07T22:40:04.003ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66572025-03-07T22:40:04.005ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
66582025-03-07T22:40:04.005ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66592025-03-07T22:40:04.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66602025-03-07T22:40:04.017ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
66612025-03-07T22:40:04.017ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 52.155352ms iteration = 38 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
66622025-03-07T22:40:04.023ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 58.961797ms iteration = 38 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
66632025-03-07T22:40:04.674ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 265 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 248706b5-0341-44e4-807a-08521679a08a response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
66642025-03-07T22:40:13.965ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 39 reason = Timeout
66652025-03-07T22:40:13.965ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66662025-03-07T22:40:13.965ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
66672025-03-07T22:40:13.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66682025-03-07T22:40:13.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66692025-03-07T22:40:13.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66702025-03-07T22:40:13.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66712025-03-07T22:40:13.967ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66722025-03-07T22:40:13.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66732025-03-07T22:40:13.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
66742025-03-07T22:40:13.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66752025-03-07T22:40:13.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66762025-03-07T22:40:13.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66772025-03-07T22:40:13.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66782025-03-07T22:40:13.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66792025-03-07T22:40:13.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
66802025-03-07T22:40:13.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66812025-03-07T22:40:13.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66822025-03-07T22:40:13.990ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
66832025-03-07T22:40:13.990ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.9131ms iteration = 39 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
66842025-03-07T22:40:14.002ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.45092ms iteration = 39 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
66852025-03-07T22:40:14.675ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 304 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 4a202fad-4287-42e7-b891-2e33f8347baf response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
66862025-03-07T22:40:19.666ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:41638
66872025-03-07T22:40:19.666ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = b2454dbe-33ed-46eb-b203-e578e355f990 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
66882025-03-07T22:40:19.666ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = b2454dbe-33ed-46eb-b203-e578e355f990 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
66892025-03-07T22:40:19.670ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3775 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = b2454dbe-33ed-46eb-b203-e578e355f990 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
66902025-03-07T22:40:19.670ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 33b95df6-b2a6-445d-9498-88041c0ae38f roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
66912025-03-07T22:40:19.670ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 33b95df6-b2a6-445d-9498-88041c0ae38f resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
66922025-03-07T22:40:19.672ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2261 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 33b95df6-b2a6-445d-9498-88041c0ae38f response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
66932025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
66942025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 40 reason = Timeout
66952025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66962025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
66972025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66982025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
66992025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 14 reason = Timeout
67002025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 14 reason = Timeout
67012025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
67022025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
67032025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 14 reason = Timeout
67042025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 14 reason = Timeout
67052025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
67062025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 14 reason = Timeout
67072025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
67082025-03-07T22:40:23.965ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
67092025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
67102025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 14 reason = Timeout
67112025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
67122025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
67132025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
67142025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
67152025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
67162025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
67172025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
67182025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 15 reason = Timeout
67192025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
67202025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
67212025-03-07T22:40:23.965ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
67222025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
67232025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67242025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
67252025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67262025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
67272025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
67282025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
67292025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
67302025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
67312025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
67322025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
67332025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
67342025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
67352025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 14 reason = Timeout
67362025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
67372025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 14 reason = Timeout
67382025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
67392025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 14 reason = Timeout
67402025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
67412025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 14 reason = Timeout
67422025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
67432025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 14 reason = Timeout
67442025-03-07T22:40:23.966ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
67452025-03-07T22:40:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
67462025-03-07T22:40:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
67472025-03-07T22:40:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
67482025-03-07T22:40:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
67492025-03-07T22:40:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
67502025-03-07T22:40:23.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67512025-03-07T22:40:23.968ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67522025-03-07T22:40:23.968ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67532025-03-07T22:40:23.968ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.11081ms iteration = 14 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
67542025-03-07T22:40:23.969ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67552025-03-07T22:40:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67562025-03-07T22:40:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
67572025-03-07T22:40:23.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67582025-03-07T22:40:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67592025-03-07T22:40:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
67602025-03-07T22:40:23.970ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
67612025-03-07T22:40:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
67622025-03-07T22:40:23.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
67632025-03-07T22:40:23.971ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
67642025-03-07T22:40:23.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
67652025-03-07T22:40:23.972ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67662025-03-07T22:40:23.973ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67672025-03-07T22:40:23.973ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67682025-03-07T22:40:23.974ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1182, tv_nsec: 327180553 } })
67692025-03-07T22:40:23.974ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
67702025-03-07T22:40:23.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
67712025-03-07T22:40:23.975ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
67722025-03-07T22:40:23.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
67732025-03-07T22:40:23.975ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
67742025-03-07T22:40:23.975ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
67752025-03-07T22:40:23.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
67762025-03-07T22:40:23.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
67772025-03-07T22:40:23.976ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
67782025-03-07T22:40:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
67792025-03-07T22:40:23.976ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1182, tv_nsec: 329429935 } })
67802025-03-07T22:40:23.977ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
67812025-03-07T22:40:23.978ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 11.157227ms iteration = 14 status = {"errors":[],"garbage_collect_requested":[]}
67822025-03-07T22:40:23.978ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 12.041556ms iteration = 14 status = {"errors":[],"finish_invoked_ok":[]}
67832025-03-07T22:40:23.980ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67842025-03-07T22:40:23.980ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
67852025-03-07T22:40:23.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67862025-03-07T22:40:23.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
67872025-03-07T22:40:23.982ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
67882025-03-07T22:40:23.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
67892025-03-07T22:40:23.982ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67902025-03-07T22:40:23.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
67912025-03-07T22:40:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
67922025-03-07T22:40:23.985ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
67932025-03-07T22:40:23.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67942025-03-07T22:40:23.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67952025-03-07T22:40:24.009ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
67962025-03-07T22:40:24.009ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 44.017153ms iteration = 40 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
67972025-03-07T22:40:24.012ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
67982025-03-07T22:40:24.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
67992025-03-07T22:40:24.015ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68002025-03-07T22:40:24.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68012025-03-07T22:40:24.017ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
68022025-03-07T22:40:24.017ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68032025-03-07T22:40:24.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
68042025-03-07T22:40:24.019ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 53.776286ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
68052025-03-07T22:40:24.019ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68062025-03-07T22:40:24.020ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
68072025-03-07T22:40:24.020ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68082025-03-07T22:40:24.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68092025-03-07T22:40:24.020ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 54.957348ms iteration = 40 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
68102025-03-07T22:40:24.021ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
68112025-03-07T22:40:24.021ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68122025-03-07T22:40:24.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68132025-03-07T22:40:24.022ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
68142025-03-07T22:40:24.022ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
68152025-03-07T22:40:24.022ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
68162025-03-07T22:40:24.022ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "95a3fd9b-d588-42b3-b724-6ba166866b05", "content-length": "26", "date": "Fri, 07 Mar 2025 22:40:24 GMT"} })
68172025-03-07T22:40:24.022ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
68182025-03-07T22:40:24.022ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 57.3034ms iteration = 14 status = 0
68192025-03-07T22:40:24.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68202025-03-07T22:40:24.023ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
68212025-03-07T22:40:24.023ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
68222025-03-07T22:40:24.023ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "936975c6-6350-4648-a927-c70a207ce3b3", "content-length": "26", "date": "Fri, 07 Mar 2025 22:40:24 GMT"} })
68232025-03-07T22:40:24.023ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
68242025-03-07T22:40:24.024ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
68252025-03-07T22:40:24.024ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
68262025-03-07T22:40:24.024ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
68272025-03-07T22:40:24.024ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
68282025-03-07T22:40:24.024ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 57.456642ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
68292025-03-07T22:40:24.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68302025-03-07T22:40:24.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68312025-03-07T22:40:24.025ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
68322025-03-07T22:40:24.025ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
68332025-03-07T22:40:24.025ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68342025-03-07T22:40:24.026ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 59.812224ms iteration = 14 status = {"errors":[],"requests_created_ok":[]}
68352025-03-07T22:40:24.027ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
68362025-03-07T22:40:24.027ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
68372025-03-07T22:40:24.027ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
68382025-03-07T22:40:24.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68392025-03-07T22:40:24.027ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68402025-03-07T22:40:24.028ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
68412025-03-07T22:40:24.028ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
68422025-03-07T22:40:24.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68432025-03-07T22:40:24.029ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
68442025-03-07T22:40:24.029ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
68452025-03-07T22:40:24.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
68462025-03-07T22:40:24.032ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1182, tv_nsec: 384726546 } })
68472025-03-07T22:40:24.032ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
68482025-03-07T22:40:24.032ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
68492025-03-07T22:40:24.032ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 67.146595ms iteration = 14 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
68502025-03-07T22:40:24.036ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68512025-03-07T22:40:24.037ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
68522025-03-07T22:40:24.037ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68532025-03-07T22:40:24.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68542025-03-07T22:40:24.041ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 75.749417ms iteration = 15 status = {"physical_disks_added":0}
68552025-03-07T22:40:24.068ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
68562025-03-07T22:40:24.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68572025-03-07T22:40:24.073ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
68582025-03-07T22:40:24.073ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 107.604794ms iteration = 14 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
68592025-03-07T22:40:24.074ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
68602025-03-07T22:40:24.074ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
68612025-03-07T22:40:24.074ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 108.567063ms iteration = 14 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
68622025-03-07T22:40:24.116ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
68632025-03-07T22:40:24.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
68642025-03-07T22:40:24.117ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 150.95968ms iteration = 14 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
68652025-03-07T22:40:24.123ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
68662025-03-07T22:40:24.124ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1c4e7f7-bccb-4230-99e7-03c623da3b2f", "content-length": "1", "date": "Fri, 07 Mar 2025 22:40:23 GMT"} })
68672025-03-07T22:40:24.127ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
68682025-03-07T22:40:24.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
68692025-03-07T22:40:24.129ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 163.927645ms iteration = 14 status = null
68702025-03-07T22:40:24.131ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 165.405219ms iteration = 14 status = {}
68712025-03-07T22:40:24.131ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
68722025-03-07T22:40:24.131ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
68732025-03-07T22:40:24.131ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff33f154-ec08-4db6-ac06-6e53a9c344d2", "content-length": "26", "date": "Fri, 07 Mar 2025 22:40:24 GMT"} })
68742025-03-07T22:40:24.131ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
68752025-03-07T22:40:24.133ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
68762025-03-07T22:40:24.170ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
68772025-03-07T22:40:24.171ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
68782025-03-07T22:40:24.172ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68792025-03-07T22:40:24.174ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
68802025-03-07T22:40:24.174ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
68812025-03-07T22:40:24.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
68822025-03-07T22:40:24.176ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68832025-03-07T22:40:24.177ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
68842025-03-07T22:40:24.177ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
68852025-03-07T22:40:24.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
68862025-03-07T22:40:24.178ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
68872025-03-07T22:40:24.178ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
68882025-03-07T22:40:24.178ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbd3a852-9b2b-414a-8df1-44ceb718b631", "content-length": "372", "date": "Fri, 07 Mar 2025 22:40:24 GMT"} })
68892025-03-07T22:40:24.178ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68902025-03-07T22:40:24.179ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68912025-03-07T22:40:24.220ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
68922025-03-07T22:40:24.220ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
68932025-03-07T22:40:24.220ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "b99bbf1f-6aed-4738-96e2-2caee0145094", "date": "Fri, 07 Mar 2025 22:40:24 GMT"} })
68942025-03-07T22:40:24.220ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68952025-03-07T22:40:24.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68962025-03-07T22:40:24.223ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68972025-03-07T22:40:24.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68982025-03-07T22:40:24.226ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68992025-03-07T22:40:24.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69002025-03-07T22:40:24.229ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
69012025-03-07T22:40:24.229ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
69022025-03-07T22:40:24.229ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "3f6f879e-bb8f-4c7a-b8cb-0293bd6bf5d0", "date": "Fri, 07 Mar 2025 22:40:24 GMT"} })
69032025-03-07T22:40:24.231ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 265.787324ms iteration = 14 status = {}
69042025-03-07T22:40:24.236ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
69052025-03-07T22:40:24.236ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "28e3ad37-080a-476f-9100-e0aed19e5fdf", "content-length": "691", "date": "Fri, 07 Mar 2025 22:40:24 GMT"} })
69062025-03-07T22:40:24.237ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
69072025-03-07T22:40:24.237ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
69082025-03-07T22:40:24.237ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
69092025-03-07T22:40:24.238ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 272.271316ms iteration = 14 status = {}
69102025-03-07T22:40:24.273ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69112025-03-07T22:40:24.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69122025-03-07T22:40:24.275ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
69132025-03-07T22:40:24.275ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69142025-03-07T22:40:24.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69152025-03-07T22:40:24.284ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
69162025-03-07T22:40:24.284ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "67a96b70-82fd-4ca6-8f32-247bdff2be99", "content-length": "134", "date": "Fri, 07 Mar 2025 22:40:23 GMT"} })
69172025-03-07T22:40:24.284ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
69182025-03-07T22:40:24.284ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
69192025-03-07T22:40:24.285ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
69202025-03-07T22:40:24.285ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
69212025-03-07T22:40:24.285ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
69222025-03-07T22:40:24.285ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
69232025-03-07T22:40:24.285ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
69242025-03-07T22:40:24.285ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
69252025-03-07T22:40:24.285ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
69262025-03-07T22:40:24.285ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
69272025-03-07T22:40:24.285ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
69282025-03-07T22:40:24.285ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69292025-03-07T22:40:24.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69302025-03-07T22:40:24.291ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
69312025-03-07T22:40:24.291ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "910d3618-1641-4dc4-bacf-04fb3b0b049f", "content-length": "2", "date": "Fri, 07 Mar 2025 22:40:23 GMT"} })
69322025-03-07T22:40:24.291ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
69332025-03-07T22:40:24.292ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "13e2c239-d076-4822-b6e5-52525a926836", "content-length": "52", "date": "Fri, 07 Mar 2025 22:40:23 GMT"} })
69342025-03-07T22:40:24.292ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
69352025-03-07T22:40:24.292ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
69362025-03-07T22:40:24.385ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "b9d8fc53-bb54-46e3-84b9-d0e92ac9a80f", "date": "Fri, 07 Mar 2025 22:40:24 GMT"} })
69372025-03-07T22:40:24.385ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
69382025-03-07T22:40:24.385ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
69392025-03-07T22:40:24.385ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "57b6c031-4128-4e21-93ff-d5861d3c1f41", "content-length": "682", "date": "Fri, 07 Mar 2025 22:40:24 GMT"} })
69402025-03-07T22:40:24.385ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69412025-03-07T22:40:24.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69422025-03-07T22:40:24.389ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69432025-03-07T22:40:24.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69442025-03-07T22:40:24.393ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69452025-03-07T22:40:24.393ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69462025-03-07T22:40:24.395ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69472025-03-07T22:40:24.395ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69482025-03-07T22:40:24.402ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 436.511315ms iteration = 14 status = {}
69492025-03-07T22:40:24.675ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 264 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 5fee3b4d-e97c-4054-add1-88a363d27926 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
69502025-03-07T22:40:33.965ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
69512025-03-07T22:40:33.965ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 41 reason = Timeout
69522025-03-07T22:40:33.965ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
69532025-03-07T22:40:33.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69542025-03-07T22:40:33.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69552025-03-07T22:40:33.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
69562025-03-07T22:40:33.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69572025-03-07T22:40:33.968ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
69582025-03-07T22:40:33.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
69592025-03-07T22:40:33.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
69602025-03-07T22:40:33.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69612025-03-07T22:40:33.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69622025-03-07T22:40:33.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69632025-03-07T22:40:33.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69642025-03-07T22:40:33.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69652025-03-07T22:40:33.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
69662025-03-07T22:40:33.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69672025-03-07T22:40:33.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69682025-03-07T22:40:33.984ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
69692025-03-07T22:40:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.65892ms iteration = 41 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
69702025-03-07T22:40:33.999ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.990286ms iteration = 41 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
69712025-03-07T22:40:34.666ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 7909238c-2ed9-4ee5-ae64-1703d970a9bf roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
69722025-03-07T22:40:34.666ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 7909238c-2ed9-4ee5-ae64-1703d970a9bf resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
69732025-03-07T22:40:34.668ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2378 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 7909238c-2ed9-4ee5-ae64-1703d970a9bf response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
69742025-03-07T22:40:34.669ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 90496342-7887-4aab-ad35-76be9c4f0d24 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
69752025-03-07T22:40:34.669ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 90496342-7887-4aab-ad35-76be9c4f0d24 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
69762025-03-07T22:40:34.671ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2862 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 90496342-7887-4aab-ad35-76be9c4f0d24 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
69772025-03-07T22:40:34.675ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 226 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 8d46e915-5dfe-45a0-816a-f6f4679c6c8b response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
69782025-03-07T22:40:43.966ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
69792025-03-07T22:40:43.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69802025-03-07T22:40:43.966ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 42 reason = Timeout
69812025-03-07T22:40:43.966ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
69822025-03-07T22:40:43.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69832025-03-07T22:40:43.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
69842025-03-07T22:40:43.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69852025-03-07T22:40:43.968ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
69862025-03-07T22:40:43.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
69872025-03-07T22:40:43.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
69882025-03-07T22:40:43.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69892025-03-07T22:40:43.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69902025-03-07T22:40:43.970ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.626565ms iteration = 42 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
69912025-03-07T22:40:43.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69922025-03-07T22:40:43.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69932025-03-07T22:40:43.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69942025-03-07T22:40:43.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
69952025-03-07T22:40:43.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69962025-03-07T22:40:43.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69972025-03-07T22:40:43.988ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
69982025-03-07T22:40:43.988ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.479976ms iteration = 42 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
69992025-03-07T22:40:44.675ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 311 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 23bcffe9-f04e-45d4-99a6-982867b4a573 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
70002025-03-07T22:40:49.666ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 651886fa-de04-41b5-b4fa-b4dcca4f7ca0 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
70012025-03-07T22:40:49.667ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 651886fa-de04-41b5-b4fa-b4dcca4f7ca0 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
70022025-03-07T22:40:49.669ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2627 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 651886fa-de04-41b5-b4fa-b4dcca4f7ca0 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
70032025-03-07T22:40:49.669ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 52ec2a6d-a9a8-4c63-9e30-df2069d55b85 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
70042025-03-07T22:40:49.669ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 52ec2a6d-a9a8-4c63-9e30-df2069d55b85 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
70052025-03-07T22:40:49.671ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2205 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41638 req_id = 52ec2a6d-a9a8-4c63-9e30-df2069d55b85 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
70062025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
70072025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 16 reason = Timeout
70082025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
70092025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70102025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
70112025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
70122025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
70132025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 8 reason = Timeout
70142025-03-07T22:40:53.966ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
70152025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
70162025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 15 reason = Timeout
70172025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
70182025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70192025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 15 reason = Timeout
70202025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
70212025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
70222025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
70232025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
70242025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
70252025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
70262025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
70272025-03-07T22:40:53.966ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
70282025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
70292025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.43\u{b5}s iteration = 9 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
70302025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
70312025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
70322025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
70332025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
70342025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
70352025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
70362025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 15 reason = Timeout
70372025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70382025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 8 reason = Timeout
70392025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
70402025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 15 reason = Timeout
70412025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
70422025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 15 reason = Timeout
70432025-03-07T22:40:53.966ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
70442025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70452025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 43 reason = Timeout
70462025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 8 reason = Timeout
70472025-03-07T22:40:53.966ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:30:53.966534539 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
70482025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 15 reason = Timeout
70492025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
70502025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
70512025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
70522025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
70532025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
70542025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
70552025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
70562025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
70572025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
70582025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 15 reason = Timeout
70592025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
70602025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 15 reason = Timeout
70612025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
70622025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 15 reason = Timeout
70632025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
70642025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 8 reason = Timeout
70652025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
70662025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 15 reason = Timeout
70672025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
70682025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 15 reason = Timeout
70692025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
70702025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70712025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
70722025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 8 reason = Timeout
70732025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
70742025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70752025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
70762025-03-07T22:40:53.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
70772025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
70782025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
70792025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
70802025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
70812025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
70822025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
70832025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70842025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
70852025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
70862025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70872025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
70882025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
70892025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
70902025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
70912025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
70922025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
70932025-03-07T22:40:53.967ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1212, tv_nsec: 319140436 } })
70942025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
70952025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
70962025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
70972025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
70982025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
70992025-03-07T22:40:53.967ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1212, tv_nsec: 319202826 } })
71002025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
71012025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.258812ms iteration = 8 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
71022025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
71032025-03-07T22:40:53.967ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.489505ms iteration = 8 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
71042025-03-07T22:40:53.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71052025-03-07T22:40:53.969ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71062025-03-07T22:40:53.969ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.345042ms iteration = 15 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
71072025-03-07T22:40:53.970ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71082025-03-07T22:40:53.970ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71092025-03-07T22:40:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
71102025-03-07T22:40:53.972ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
71112025-03-07T22:40:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
71122025-03-07T22:40:53.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71132025-03-07T22:40:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
71142025-03-07T22:40:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71152025-03-07T22:40:53.972ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
71162025-03-07T22:40:53.973ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
71172025-03-07T22:40:53.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
71182025-03-07T22:40:53.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
71192025-03-07T22:40:53.973ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
71202025-03-07T22:40:53.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71212025-03-07T22:40:53.974ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
71222025-03-07T22:40:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
71232025-03-07T22:40:53.974ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71242025-03-07T22:40:53.974ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
71252025-03-07T22:40:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
71262025-03-07T22:40:53.975ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71272025-03-07T22:40:53.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
71282025-03-07T22:40:53.976ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
71292025-03-07T22:40:53.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
71302025-03-07T22:40:53.977ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
71312025-03-07T22:40:53.977ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
71322025-03-07T22:40:53.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
71332025-03-07T22:40:53.978ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71342025-03-07T22:40:53.978ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71352025-03-07T22:40:53.979ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71362025-03-07T22:40:53.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71372025-03-07T22:40:53.980ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
71382025-03-07T22:40:53.980ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71392025-03-07T22:40:53.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71402025-03-07T22:40:53.980ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
71412025-03-07T22:40:53.980ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
71422025-03-07T22:40:53.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
71432025-03-07T22:40:53.982ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:30:53.966534539 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
71442025-03-07T22:40:53.982ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 16.385778ms iteration = 8 status = {"expiration":"2025-03-07T22:30:53.966534539Z","pruned":[]}
71452025-03-07T22:40:53.982ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1212, tv_nsec: 334877517 } })
71462025-03-07T22:40:53.983ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
71472025-03-07T22:40:53.984ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1212, tv_nsec: 336041068 } })
71482025-03-07T22:40:53.984ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
71492025-03-07T22:40:53.985ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 18.423827ms iteration = 8 status = {"errors":[],"found_port_ok":[]}
71502025-03-07T22:40:53.986ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
71512025-03-07T22:40:53.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
71522025-03-07T22:40:53.987ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71532025-03-07T22:40:53.987ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
71542025-03-07T22:40:53.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71552025-03-07T22:40:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
71562025-03-07T22:40:53.988ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
71572025-03-07T22:40:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
71582025-03-07T22:40:53.988ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
71592025-03-07T22:40:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71602025-03-07T22:40:53.989ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71612025-03-07T22:40:53.991ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71622025-03-07T22:40:53.992ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
71632025-03-07T22:40:53.993ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
71642025-03-07T22:40:53.993ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
71652025-03-07T22:40:53.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
71662025-03-07T22:40:53.998ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1212, tv_nsec: 350648988 } })
71672025-03-07T22:40:53.999ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
71682025-03-07T22:40:54.001ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71692025-03-07T22:40:54.006ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
71702025-03-07T22:40:54.006ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71712025-03-07T22:40:54.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71722025-03-07T22:40:54.010ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71732025-03-07T22:40:54.012ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
71742025-03-07T22:40:54.013ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
71752025-03-07T22:40:54.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
71762025-03-07T22:40:54.015ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71772025-03-07T22:40:54.015ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
71782025-03-07T22:40:54.015ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
71792025-03-07T22:40:54.015ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
71802025-03-07T22:40:54.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
71812025-03-07T22:40:54.018ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
71822025-03-07T22:40:54.018ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
71832025-03-07T22:40:54.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
71842025-03-07T22:40:54.022ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
71852025-03-07T22:40:54.022ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
71862025-03-07T22:40:54.022ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 55.353801ms iteration = 8 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
71872025-03-07T22:40:54.026ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71882025-03-07T22:40:54.028ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
71892025-03-07T22:40:54.029ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
71902025-03-07T22:40:54.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
71912025-03-07T22:40:54.029ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
71922025-03-07T22:40:54.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
71932025-03-07T22:40:54.031ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71942025-03-07T22:40:54.031ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 65.195906ms iteration = 15 status = {"errors":[],"finish_invoked_ok":[]}
71952025-03-07T22:40:54.033ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
71962025-03-07T22:40:54.033ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
71972025-03-07T22:40:54.033ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
71982025-03-07T22:40:54.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
71992025-03-07T22:40:54.034ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 68.090983ms iteration = 15 status = {"errors":[],"garbage_collect_requested":[]}
72002025-03-07T22:40:54.036ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
72012025-03-07T22:40:54.037ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 70.693699ms iteration = 43 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
72022025-03-07T22:40:54.037ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
72032025-03-07T22:40:54.037ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
72042025-03-07T22:40:54.037ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
72052025-03-07T22:40:54.037ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
72062025-03-07T22:40:54.037ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
72072025-03-07T22:40:54.037ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 71.240724ms iteration = 43 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
72082025-03-07T22:40:54.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
72092025-03-07T22:40:54.038ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72102025-03-07T22:40:54.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72112025-03-07T22:40:54.038ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
72122025-03-07T22:40:54.039ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 72.756079ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
72132025-03-07T22:40:54.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
72142025-03-07T22:40:54.039ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 73.141523ms iteration = 15 status = {"errors":[],"requests_created_ok":[]}
72152025-03-07T22:40:54.040ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72162025-03-07T22:40:54.041ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
72172025-03-07T22:40:54.041ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
72182025-03-07T22:40:54.041ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 75.047731ms iteration = 8 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
72192025-03-07T22:40:54.041ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72202025-03-07T22:40:54.041ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72212025-03-07T22:40:54.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72222025-03-07T22:40:54.043ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
72232025-03-07T22:40:54.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
72242025-03-07T22:40:54.044ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
72252025-03-07T22:40:54.044ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
72262025-03-07T22:40:54.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
72272025-03-07T22:40:54.045ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
72282025-03-07T22:40:54.045ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
72292025-03-07T22:40:54.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
72302025-03-07T22:40:54.046ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72312025-03-07T22:40:54.048ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
72322025-03-07T22:40:54.048ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 81.877456ms iteration = 15 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
72332025-03-07T22:40:54.049ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
72342025-03-07T22:40:54.049ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
72352025-03-07T22:40:54.049ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
72362025-03-07T22:40:54.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
72372025-03-07T22:40:54.051ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72382025-03-07T22:40:54.051ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
72392025-03-07T22:40:54.051ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72402025-03-07T22:40:54.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72412025-03-07T22:40:54.054ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
72422025-03-07T22:40:54.054ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
72432025-03-07T22:40:54.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
72442025-03-07T22:40:54.054ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 88.263048ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
72452025-03-07T22:40:54.057ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 90.936313ms iteration = 15 status = 0
72462025-03-07T22:40:54.063ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
72472025-03-07T22:40:54.063ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
72482025-03-07T22:40:54.063ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-07 22:34:17.208516 UTC dns_group = internal generation = 1
72492025-03-07T22:40:54.063ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 96.83522ms iteration = 8 status = {"generation":1}
72502025-03-07T22:40:54.075ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72512025-03-07T22:40:54.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72522025-03-07T22:40:54.083ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
72532025-03-07T22:40:54.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
72542025-03-07T22:40:54.091ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72552025-03-07T22:40:54.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72562025-03-07T22:40:54.093ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
72572025-03-07T22:40:54.093ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
72582025-03-07T22:40:54.093ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
72592025-03-07T22:40:54.093ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
72602025-03-07T22:40:54.093ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72612025-03-07T22:40:54.093ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
72622025-03-07T22:40:54.094ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 128.636016ms iteration = 16 status = {"physical_disks_added":0}
72632025-03-07T22:40:54.096ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
72642025-03-07T22:40:54.096ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
72652025-03-07T22:40:54.096ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 129.860487ms iteration = 15 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
72662025-03-07T22:40:54.096ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
72672025-03-07T22:40:54.096ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72682025-03-07T22:40:54.096ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72692025-03-07T22:40:54.098ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72702025-03-07T22:40:54.100ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
72712025-03-07T22:40:54.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
72722025-03-07T22:40:54.101ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
72732025-03-07T22:40:54.101ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72742025-03-07T22:40:54.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72752025-03-07T22:40:54.106ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
72762025-03-07T22:40:54.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
72772025-03-07T22:40:54.108ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
72782025-03-07T22:40:54.124ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
72792025-03-07T22:40:54.134ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) }
72802025-03-07T22:40:54.135ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 168.60096ms iteration = 8 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
72812025-03-07T22:40:54.143ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
72822025-03-07T22:40:54.143ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
72832025-03-07T22:40:54.143ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1b46b32-8d02-454b-8189-980e9c498781", "content-length": "26", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} })
72842025-03-07T22:40:54.143ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
72852025-03-07T22:40:54.146ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
72862025-03-07T22:40:54.158ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72872025-03-07T22:40:54.159ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 192.451889ms iteration = 15 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
72882025-03-07T22:40:54.160ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
72892025-03-07T22:40:54.160ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
72902025-03-07T22:40:54.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
72912025-03-07T22:40:54.161ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72922025-03-07T22:40:54.163ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
72932025-03-07T22:40:54.163ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
72942025-03-07T22:40:54.163ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
72952025-03-07T22:40:54.163ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
72962025-03-07T22:40:54.163ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
72972025-03-07T22:40:54.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
72982025-03-07T22:40:54.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
72992025-03-07T22:40:54.185ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
73002025-03-07T22:40:54.185ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 218.933673ms iteration = 8 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
73012025-03-07T22:40:54.189ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
73022025-03-07T22:40:54.189ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
73032025-03-07T22:40:54.190ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdcd959a-89e0-4d70-882c-3c6dc66b46d5", "content-length": "26", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} })
73042025-03-07T22:40:54.190ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
73052025-03-07T22:40:54.191ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
73062025-03-07T22:40:54.233ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
73072025-03-07T22:40:54.233ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
73082025-03-07T22:40:54.233ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cd6ea4c-dd42-4724-953b-9725f745828b", "content-length": "26", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} })
73092025-03-07T22:40:54.233ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
73102025-03-07T22:40:54.235ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
73112025-03-07T22:40:54.294ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
73122025-03-07T22:40:54.294ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bc4545e-216f-4297-be04-e57b1d04ef31", "content-length": "691", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} })
73132025-03-07T22:40:54.295ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
73142025-03-07T22:40:54.295ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
73152025-03-07T22:40:54.295ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
73162025-03-07T22:40:54.296ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 330.245213ms iteration = 15 status = {}
73172025-03-07T22:40:54.328ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
73182025-03-07T22:40:54.357ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
73192025-03-07T22:40:54.357ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
73202025-03-07T22:40:54.357ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "205f62a8-1829-4ecd-8650-a7ebf292af5c", "content-length": "372", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} })
73212025-03-07T22:40:54.358ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73222025-03-07T22:40:54.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73232025-03-07T22:40:54.360ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
73242025-03-07T22:40:54.360ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 393.510752ms iteration = 15 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
73252025-03-07T22:40:54.360ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73262025-03-07T22:40:54.363ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
73272025-03-07T22:40:54.363ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
73282025-03-07T22:40:54.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
73292025-03-07T22:40:54.365ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
73302025-03-07T22:40:54.365ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
73312025-03-07T22:40:54.365ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-07 22:34:17.797135 UTC dns_group = external generation = 2
73322025-03-07T22:40:54.365ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 399.222917ms iteration = 8 status = {"generation":2}
73332025-03-07T22:40:54.373ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 406.605077ms iteration = 15 status = {}
73342025-03-07T22:40:54.401ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
73352025-03-07T22:40:54.401ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
73362025-03-07T22:40:54.401ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "556194cf-df81-45fa-a942-8ef3ddd8aad1", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} })
73372025-03-07T22:40:54.401ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73382025-03-07T22:40:54.401ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73392025-03-07T22:40:54.403ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73402025-03-07T22:40:54.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73412025-03-07T22:40:54.406ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73422025-03-07T22:40:54.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73432025-03-07T22:40:54.415ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
73442025-03-07T22:40:54.415ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
73452025-03-07T22:40:54.415ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "f9f0f44b-3b13-4895-9777-3e02d668f45d", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} })
73462025-03-07T22:40:54.417ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 451.319387ms iteration = 15 status = {}
73472025-03-07T22:40:54.429ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
73482025-03-07T22:40:54.429ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3124c76-0f9c-45bd-9b88-56f067b1c19a", "content-length": "1", "date": "Fri, 07 Mar 2025 22:40:53 GMT"} })
73492025-03-07T22:40:54.430ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
73502025-03-07T22:40:54.430ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "9478fb80-1206-48a6-9698-73106a78dbe7", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
73512025-03-07T22:40:54.430ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
73522025-03-07T22:40:54.431ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
73532025-03-07T22:40:54.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
73542025-03-07T22:40:54.433ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "fa28eb59-3d0d-4797-a40f-4fa1aaabacd4", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
73552025-03-07T22:40:54.433ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
73562025-03-07T22:40:54.434ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 467.918496ms iteration = 15 status = null
73572025-03-07T22:40:54.435ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
73582025-03-07T22:40:54.435ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 469.464201ms iteration = 10 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
73592025-03-07T22:40:54.448ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
73602025-03-07T22:40:54.449ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "48cf54b5-3a39-4d28-b7f9-0802ac29fac9", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
73612025-03-07T22:40:54.449ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
73622025-03-07T22:40:54.450ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "ca45f255-05e3-4fbf-aff7-15005eff5aea", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
73632025-03-07T22:40:54.450ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
73642025-03-07T22:40:54.452ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "1fd67949-261c-43c1-9219-8c61cbb25214", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
73652025-03-07T22:40:54.452ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
73662025-03-07T22:40:54.454ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
73672025-03-07T22:40:54.454ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 487.682707ms iteration = 10 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
73682025-03-07T22:40:54.518ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73692025-03-07T22:40:54.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73702025-03-07T22:40:54.521ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
73712025-03-07T22:40:54.521ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73722025-03-07T22:40:54.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73732025-03-07T22:40:54.539ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
73742025-03-07T22:40:54.540ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "d486fa71-e474-4624-b7ba-9287d56ae2b8", "content-length": "134", "date": "Fri, 07 Mar 2025 22:40:53 GMT"} })
73752025-03-07T22:40:54.540ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
73762025-03-07T22:40:54.540ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
73772025-03-07T22:40:54.540ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
73782025-03-07T22:40:54.540ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
73792025-03-07T22:40:54.540ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
73802025-03-07T22:40:54.540ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
73812025-03-07T22:40:54.540ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
73822025-03-07T22:40:54.540ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
73832025-03-07T22:40:54.541ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
73842025-03-07T22:40:54.541ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
73852025-03-07T22:40:54.541ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
73862025-03-07T22:40:54.541ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73872025-03-07T22:40:54.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73882025-03-07T22:40:54.543ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
73892025-03-07T22:40:54.543ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7315f02-1d41-4fac-b5c0-352877cd7d60", "content-length": "2", "date": "Fri, 07 Mar 2025 22:40:53 GMT"} })
73902025-03-07T22:40:54.543ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
73912025-03-07T22:40:54.544ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "312a7f29-0bb1-49a9-ae15-9616e888a794", "content-length": "52", "date": "Fri, 07 Mar 2025 22:40:53 GMT"} })
73922025-03-07T22:40:54.544ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
73932025-03-07T22:40:54.544ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
73942025-03-07T22:40:54.666ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "64931433-8445-4de4-8177-7aec182f1f23", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} })
73952025-03-07T22:40:54.666ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
73962025-03-07T22:40:54.666ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
73972025-03-07T22:40:54.666ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "8998bb25-e624-44a6-9983-8e66eb279f18", "content-length": "682", "date": "Fri, 07 Mar 2025 22:40:54 GMT"} })
73982025-03-07T22:40:54.666ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73992025-03-07T22:40:54.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74002025-03-07T22:40:54.671ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74012025-03-07T22:40:54.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74022025-03-07T22:40:54.675ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74032025-03-07T22:40:54.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74042025-03-07T22:40:54.676ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 270 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 755be54f-6df1-4884-982c-68de8b452036 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
74052025-03-07T22:40:54.678ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74062025-03-07T22:40:54.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74072025-03-07T22:40:54.685ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 718.682397ms iteration = 15 status = {}
74082025-03-07T22:41:03.966ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 44 reason = Timeout
74092025-03-07T22:41:03.966ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74102025-03-07T22:41:03.966ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
74112025-03-07T22:41:03.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74122025-03-07T22:41:03.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74132025-03-07T22:41:03.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74142025-03-07T22:41:03.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74152025-03-07T22:41:03.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74162025-03-07T22:41:03.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74172025-03-07T22:41:03.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74182025-03-07T22:41:03.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74192025-03-07T22:41:03.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74202025-03-07T22:41:03.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74212025-03-07T22:41:03.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
74222025-03-07T22:41:03.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74232025-03-07T22:41:03.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74242025-03-07T22:41:03.989ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
74252025-03-07T22:41:03.989ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.396145ms iteration = 44 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
74262025-03-07T22:41:03.994ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74272025-03-07T22:41:03.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74282025-03-07T22:41:03.995ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 29.444543ms iteration = 44 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
74292025-03-07T22:41:04.676ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 308 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 650b1a7d-af55-4f8d-b5b1-a7040c4ecb29 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
74302025-03-07T22:41:13.966ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
74312025-03-07T22:41:13.966ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 45 reason = Timeout
74322025-03-07T22:41:13.966ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74332025-03-07T22:41:13.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74342025-03-07T22:41:13.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74352025-03-07T22:41:13.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74362025-03-07T22:41:13.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74372025-03-07T22:41:13.968ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74382025-03-07T22:41:13.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74392025-03-07T22:41:13.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74402025-03-07T22:41:13.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74412025-03-07T22:41:13.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74422025-03-07T22:41:13.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74432025-03-07T22:41:13.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74442025-03-07T22:41:13.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74452025-03-07T22:41:13.977ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
74462025-03-07T22:41:13.977ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74472025-03-07T22:41:13.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74482025-03-07T22:41:13.989ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
74492025-03-07T22:41:13.989ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.574117ms iteration = 45 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
74502025-03-07T22:41:14.001ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.27536ms iteration = 45 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
74512025-03-07T22:41:14.677ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 300 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 2e3c7e41-46d0-4384-b1ad-f610ef916bc2 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
74522025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
74532025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 16 reason = Timeout
74542025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
74552025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
74562025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
74572025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
74582025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
74592025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 16 reason = Timeout
74602025-03-07T22:41:23.966ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
74612025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74622025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 46 reason = Timeout
74632025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
74642025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74652025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 16 reason = Timeout
74662025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 16 reason = Timeout
74672025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74682025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 17 reason = Timeout
74692025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
74702025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
74712025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
74722025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
74732025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
74742025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
74752025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 16 reason = Timeout
74762025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
74772025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 16 reason = Timeout
74782025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
74792025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
74802025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
74812025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
74822025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
74832025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
74842025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
74852025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
74862025-03-07T22:41:23.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74872025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74882025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74892025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
74902025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
74912025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
74922025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
74932025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
74942025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 16 reason = Timeout
74952025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 16 reason = Timeout
74962025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
74972025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
74982025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 16 reason = Timeout
74992025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
75002025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 16 reason = Timeout
75012025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
75022025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 16 reason = Timeout
75032025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
75042025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
75052025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
75062025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
75072025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
75082025-03-07T22:41:23.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
75092025-03-07T22:41:23.968ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75102025-03-07T22:41:23.969ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75112025-03-07T22:41:23.969ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.0638ms iteration = 16 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
75122025-03-07T22:41:23.970ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
75132025-03-07T22:41:23.970ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75142025-03-07T22:41:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75152025-03-07T22:41:23.971ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1242, tv_nsec: 322485688 } })
75162025-03-07T22:41:23.971ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
75172025-03-07T22:41:23.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
75182025-03-07T22:41:23.971ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75192025-03-07T22:41:23.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75202025-03-07T22:41:23.974ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75212025-03-07T22:41:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
75222025-03-07T22:41:23.976ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75232025-03-07T22:41:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75242025-03-07T22:41:23.977ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1242, tv_nsec: 328388485 } })
75252025-03-07T22:41:23.977ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
75262025-03-07T22:41:23.984ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75272025-03-07T22:41:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75282025-03-07T22:41:23.986ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75292025-03-07T22:41:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
75302025-03-07T22:41:23.988ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75312025-03-07T22:41:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75322025-03-07T22:41:23.990ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
75332025-03-07T22:41:23.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
75342025-03-07T22:41:24.002ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75352025-03-07T22:41:24.004ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
75362025-03-07T22:41:24.004ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
75372025-03-07T22:41:24.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
75382025-03-07T22:41:24.010ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75392025-03-07T22:41:24.012ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
75402025-03-07T22:41:24.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
75412025-03-07T22:41:24.012ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
75422025-03-07T22:41:24.012ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75432025-03-07T22:41:24.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75442025-03-07T22:41:24.013ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 46.152904ms iteration = 16 status = {"errors":[],"garbage_collect_requested":[]}
75452025-03-07T22:41:24.014ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 47.87684ms iteration = 46 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
75462025-03-07T22:41:24.015ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
75472025-03-07T22:41:24.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
75482025-03-07T22:41:24.015ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75492025-03-07T22:41:24.015ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
75502025-03-07T22:41:24.015ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
75512025-03-07T22:41:24.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75522025-03-07T22:41:24.015ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "13e7c8fc-b7e9-4afd-b26d-a7a7b746515d", "content-length": "26", "date": "Fri, 07 Mar 2025 22:41:24 GMT"} })
75532025-03-07T22:41:24.016ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
75542025-03-07T22:41:24.016ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 48.983701ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
75552025-03-07T22:41:24.018ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
75562025-03-07T22:41:24.018ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
75572025-03-07T22:41:24.018ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
75582025-03-07T22:41:24.018ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f1c1257-817e-48fc-9831-46a5ee770e2d", "content-length": "26", "date": "Fri, 07 Mar 2025 22:41:24 GMT"} })
75592025-03-07T22:41:24.018ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
75602025-03-07T22:41:24.019ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
75612025-03-07T22:41:24.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
75622025-03-07T22:41:24.021ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
75632025-03-07T22:41:24.021ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
75642025-03-07T22:41:24.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
75652025-03-07T22:41:24.023ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
75662025-03-07T22:41:24.023ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
75672025-03-07T22:41:24.023ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
75682025-03-07T22:41:24.023ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75692025-03-07T22:41:24.023ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75702025-03-07T22:41:24.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
75712025-03-07T22:41:24.024ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 56.920737ms iteration = 16 status = {"errors":[],"finish_invoked_ok":[]}
75722025-03-07T22:41:24.025ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
75732025-03-07T22:41:24.025ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75742025-03-07T22:41:24.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75752025-03-07T22:41:24.025ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
75762025-03-07T22:41:24.025ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
75772025-03-07T22:41:24.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
75782025-03-07T22:41:24.026ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75792025-03-07T22:41:24.028ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 60.700763ms iteration = 16 status = {"errors":[],"requests_created_ok":[]}
75802025-03-07T22:41:24.028ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
75812025-03-07T22:41:24.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
75822025-03-07T22:41:24.029ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
75832025-03-07T22:41:24.029ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75842025-03-07T22:41:24.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75852025-03-07T22:41:24.035ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75862025-03-07T22:41:24.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75872025-03-07T22:41:24.037ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75882025-03-07T22:41:24.037ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1242, tv_nsec: 389077897 } })
75892025-03-07T22:41:24.038ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75902025-03-07T22:41:24.038ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
75912025-03-07T22:41:24.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75922025-03-07T22:41:24.039ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
75932025-03-07T22:41:24.039ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75942025-03-07T22:41:24.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75952025-03-07T22:41:24.041ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
75962025-03-07T22:41:24.041ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 74.685038ms iteration = 16 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
75972025-03-07T22:41:24.042ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
75982025-03-07T22:41:24.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
75992025-03-07T22:41:24.052ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
76002025-03-07T22:41:24.052ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 85.766254ms iteration = 16 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
76012025-03-07T22:41:24.062ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76022025-03-07T22:41:24.065ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
76032025-03-07T22:41:24.065ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76042025-03-07T22:41:24.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76052025-03-07T22:41:24.069ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76062025-03-07T22:41:24.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76072025-03-07T22:41:24.073ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76082025-03-07T22:41:24.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76092025-03-07T22:41:24.078ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76102025-03-07T22:41:24.081ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
76112025-03-07T22:41:24.081ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
76122025-03-07T22:41:24.081ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
76132025-03-07T22:41:24.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
76142025-03-07T22:41:24.082ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
76152025-03-07T22:41:24.082ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76162025-03-07T22:41:24.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76172025-03-07T22:41:24.084ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 116.412609ms iteration = 16 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
76182025-03-07T22:41:24.086ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 120.401746ms iteration = 16 status = 0
76192025-03-07T22:41:24.089ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
76202025-03-07T22:41:24.089ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 123.555847ms iteration = 46 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
76212025-03-07T22:41:24.090ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 124.102782ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
76222025-03-07T22:41:24.099ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76232025-03-07T22:41:24.101ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
76242025-03-07T22:41:24.101ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
76252025-03-07T22:41:24.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
76262025-03-07T22:41:24.103ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76272025-03-07T22:41:24.105ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
76282025-03-07T22:41:24.105ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
76292025-03-07T22:41:24.105ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
76302025-03-07T22:41:24.110ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 144.491299ms iteration = 16 status = {}
76312025-03-07T22:41:24.129ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
76322025-03-07T22:41:24.129ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
76332025-03-07T22:41:24.129ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d247259b-0cfa-468b-b84c-b161f3c9efb9", "content-length": "26", "date": "Fri, 07 Mar 2025 22:41:24 GMT"} })
76342025-03-07T22:41:24.129ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
76352025-03-07T22:41:24.130ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
76362025-03-07T22:41:24.138ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
76372025-03-07T22:41:24.139ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "17b02d70-eb52-45dd-b5b8-bc56a3831961", "content-length": "1", "date": "Fri, 07 Mar 2025 22:41:23 GMT"} })
76382025-03-07T22:41:24.141ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
76392025-03-07T22:41:24.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
76402025-03-07T22:41:24.142ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
76412025-03-07T22:41:24.142ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
76422025-03-07T22:41:24.142ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 175.672058ms iteration = 16 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
76432025-03-07T22:41:24.143ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 177.341793ms iteration = 16 status = null
76442025-03-07T22:41:24.157ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 190.876104ms iteration = 17 status = {"physical_disks_added":0}
76452025-03-07T22:41:24.189ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
76462025-03-07T22:41:24.189ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
76472025-03-07T22:41:24.190ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dfc03f6-c60d-4b3a-8fed-0eda7e06bc2d", "content-length": "372", "date": "Fri, 07 Mar 2025 22:41:24 GMT"} })
76482025-03-07T22:41:24.190ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76492025-03-07T22:41:24.191ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76502025-03-07T22:41:24.209ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
76512025-03-07T22:41:24.209ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
76522025-03-07T22:41:24.209ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "e0746105-db0f-436f-b4a4-1988c7e96b61", "date": "Fri, 07 Mar 2025 22:41:24 GMT"} })
76532025-03-07T22:41:24.209ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76542025-03-07T22:41:24.209ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76552025-03-07T22:41:24.213ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76562025-03-07T22:41:24.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76572025-03-07T22:41:24.259ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
76582025-03-07T22:41:24.260ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b0f6ff8-f431-4026-8a9a-94c1b765545e", "content-length": "691", "date": "Fri, 07 Mar 2025 22:41:24 GMT"} })
76592025-03-07T22:41:24.260ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
76602025-03-07T22:41:24.260ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
76612025-03-07T22:41:24.260ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
76622025-03-07T22:41:24.262ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 295.787572ms iteration = 16 status = {}
76632025-03-07T22:41:24.264ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76642025-03-07T22:41:24.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76652025-03-07T22:41:24.268ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
76662025-03-07T22:41:24.268ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
76672025-03-07T22:41:24.269ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "8a0d5891-8079-4c23-9016-a9915244501f", "date": "Fri, 07 Mar 2025 22:41:24 GMT"} })
76682025-03-07T22:41:24.271ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 305.166472ms iteration = 16 status = {}
76692025-03-07T22:41:24.304ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76702025-03-07T22:41:24.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
76712025-03-07T22:41:24.307ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
76722025-03-07T22:41:24.307ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76732025-03-07T22:41:24.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
76742025-03-07T22:41:24.318ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
76752025-03-07T22:41:24.319ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "6aa9c970-7609-45c1-b0c4-b4ffa424094d", "content-length": "134", "date": "Fri, 07 Mar 2025 22:41:23 GMT"} })
76762025-03-07T22:41:24.319ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
76772025-03-07T22:41:24.319ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
76782025-03-07T22:41:24.319ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
76792025-03-07T22:41:24.319ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
76802025-03-07T22:41:24.319ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
76812025-03-07T22:41:24.319ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
76822025-03-07T22:41:24.319ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
76832025-03-07T22:41:24.319ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
76842025-03-07T22:41:24.319ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
76852025-03-07T22:41:24.319ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
76862025-03-07T22:41:24.319ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
76872025-03-07T22:41:24.319ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76882025-03-07T22:41:24.320ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
76892025-03-07T22:41:24.322ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
76902025-03-07T22:41:24.322ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "3837bca8-b12e-44e0-8486-17ead35099b7", "content-length": "2", "date": "Fri, 07 Mar 2025 22:41:23 GMT"} })
76912025-03-07T22:41:24.322ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
76922025-03-07T22:41:24.322ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "d55babd7-beb7-44c5-a47a-0b72a288d1c4", "content-length": "52", "date": "Fri, 07 Mar 2025 22:41:23 GMT"} })
76932025-03-07T22:41:24.322ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
76942025-03-07T22:41:24.322ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
76952025-03-07T22:41:24.422ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "90b72377-81c4-4741-8136-df991eece5f2", "date": "Fri, 07 Mar 2025 22:41:24 GMT"} })
76962025-03-07T22:41:24.422ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
76972025-03-07T22:41:24.422ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
76982025-03-07T22:41:24.422ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "87d7b10f-1ba3-46d1-941d-252d18bfc808", "content-length": "682", "date": "Fri, 07 Mar 2025 22:41:24 GMT"} })
76992025-03-07T22:41:24.423ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77002025-03-07T22:41:24.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77012025-03-07T22:41:24.433ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77022025-03-07T22:41:24.433ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77032025-03-07T22:41:24.437ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77042025-03-07T22:41:24.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77052025-03-07T22:41:24.439ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77062025-03-07T22:41:24.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77072025-03-07T22:41:24.448ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 482.250114ms iteration = 16 status = {}
77082025-03-07T22:41:24.676ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 273 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 9702391e-5aaa-4452-a29f-21c5c619f3b0 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
77092025-03-07T22:41:33.966ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
77102025-03-07T22:41:33.967ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 47 reason = Timeout
77112025-03-07T22:41:33.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77122025-03-07T22:41:33.967ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77132025-03-07T22:41:33.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77142025-03-07T22:41:33.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77152025-03-07T22:41:33.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77162025-03-07T22:41:33.969ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77172025-03-07T22:41:33.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77182025-03-07T22:41:33.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
77192025-03-07T22:41:33.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77202025-03-07T22:41:33.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77212025-03-07T22:41:33.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77222025-03-07T22:41:33.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77232025-03-07T22:41:33.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77242025-03-07T22:41:33.977ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
77252025-03-07T22:41:33.977ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77262025-03-07T22:41:33.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77272025-03-07T22:41:33.986ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
77282025-03-07T22:41:33.986ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.74278ms iteration = 47 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
77292025-03-07T22:41:33.999ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.655684ms iteration = 47 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
77302025-03-07T22:41:34.677ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 274 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 2a091532-10bf-4d92-8f1e-7813050695a9 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
77312025-03-07T22:41:43.966ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
77322025-03-07T22:41:43.966ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 48 reason = Timeout
77332025-03-07T22:41:43.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77342025-03-07T22:41:43.966ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77352025-03-07T22:41:43.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77362025-03-07T22:41:43.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77372025-03-07T22:41:43.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77382025-03-07T22:41:43.969ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77392025-03-07T22:41:43.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77402025-03-07T22:41:43.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
77412025-03-07T22:41:43.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77422025-03-07T22:41:43.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77432025-03-07T22:41:43.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77442025-03-07T22:41:43.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77452025-03-07T22:41:43.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77462025-03-07T22:41:43.978ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
77472025-03-07T22:41:43.978ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77482025-03-07T22:41:43.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77492025-03-07T22:41:43.994ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
77502025-03-07T22:41:43.994ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.670166ms iteration = 48 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
77512025-03-07T22:41:44.004ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.546531ms iteration = 48 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
77522025-03-07T22:41:44.677ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 295 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 803eff71-8522-440c-899b-99c8092bdf02 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
77532025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
77542025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 18 reason = Timeout
77552025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77562025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
77572025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
77582025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 17 reason = Timeout
77592025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
77602025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
77612025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 17 reason = Timeout
77622025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 17 reason = Timeout
77632025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
77642025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
77652025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
77662025-03-07T22:41:53.967ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
77672025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
77682025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 12.28\u{b5}s iteration = 10 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
77692025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 17 reason = Timeout
77702025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 17 reason = Timeout
77712025-03-07T22:41:53.967ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
77722025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 17 reason = Timeout
77732025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
77742025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77752025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
77762025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 9 reason = Timeout
77772025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77782025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
77792025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
77802025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 17 reason = Timeout
77812025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
77822025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
77832025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 9 reason = Timeout
77842025-03-07T22:41:53.967ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
77852025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
77862025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 49 reason = Timeout
77872025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77882025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 17 reason = Timeout
77892025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 9 reason = Timeout
77902025-03-07T22:41:53.967ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
77912025-03-07T22:41:53.968ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:31:53.967904611 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
77922025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
77932025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 17 reason = Timeout
77942025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
77952025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
77962025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
77972025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 9 reason = Timeout
77982025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
77992025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 9 reason = Timeout
78002025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
78012025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 17 reason = Timeout
78022025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
78032025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
78042025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
78052025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 17 reason = Timeout
78062025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
78072025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
78082025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
78092025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
78102025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
78112025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
78122025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
78132025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
78142025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
78152025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
78162025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
78172025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
78182025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
78192025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
78202025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
78212025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78222025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
78232025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
78242025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
78252025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
78262025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
78272025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78282025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78292025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
78302025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
78312025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
78322025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
78332025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
78342025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
78352025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
78362025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
78372025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
78382025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
78392025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
78402025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
78412025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
78422025-03-07T22:41:53.968ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1272, tv_nsec: 319408458 } })
78432025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
78442025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
78452025-03-07T22:41:53.968ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1272, tv_nsec: 319445488 } })
78462025-03-07T22:41:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
78472025-03-07T22:41:53.969ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
78482025-03-07T22:41:53.969ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.384853ms iteration = 9 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
78492025-03-07T22:41:53.969ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
78502025-03-07T22:41:53.969ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.666656ms iteration = 9 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
78512025-03-07T22:41:53.969ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78522025-03-07T22:41:53.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78532025-03-07T22:41:53.971ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.861757ms iteration = 17 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
78542025-03-07T22:41:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
78552025-03-07T22:41:53.972ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
78562025-03-07T22:41:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
78572025-03-07T22:41:53.973ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
78582025-03-07T22:41:53.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
78592025-03-07T22:41:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
78602025-03-07T22:41:53.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78612025-03-07T22:41:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78622025-03-07T22:41:53.974ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78632025-03-07T22:41:53.974ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78642025-03-07T22:41:53.975ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 7.811104ms iteration = 17 status = {"errors":[],"garbage_collect_requested":[]}
78652025-03-07T22:41:53.976ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78662025-03-07T22:41:53.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
78672025-03-07T22:41:53.976ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
78682025-03-07T22:41:53.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
78692025-03-07T22:41:53.977ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
78702025-03-07T22:41:53.977ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
78712025-03-07T22:41:53.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
78722025-03-07T22:41:53.978ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
78732025-03-07T22:41:53.978ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78742025-03-07T22:41:53.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78752025-03-07T22:41:53.979ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78762025-03-07T22:41:53.980ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78772025-03-07T22:41:53.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
78782025-03-07T22:41:53.981ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
78792025-03-07T22:41:53.981ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
78802025-03-07T22:41:53.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
78812025-03-07T22:41:53.981ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
78822025-03-07T22:41:53.981ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 13.708992ms iteration = 9 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
78832025-03-07T22:41:53.981ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1272, tv_nsec: 332431913 } })
78842025-03-07T22:41:53.981ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
78852025-03-07T22:41:53.982ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
78862025-03-07T22:41:53.982ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
78872025-03-07T22:41:53.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
78882025-03-07T22:41:53.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
78892025-03-07T22:41:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
78902025-03-07T22:41:53.984ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 16.069444ms iteration = 17 status = {"errors":[],"finish_invoked_ok":[]}
78912025-03-07T22:41:53.985ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
78922025-03-07T22:41:53.985ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:31:53.967904611 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
78932025-03-07T22:41:53.985ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 17.868672ms iteration = 9 status = {"expiration":"2025-03-07T22:31:53.967904611Z","pruned":[]}
78942025-03-07T22:41:53.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
78952025-03-07T22:41:53.986ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
78962025-03-07T22:41:53.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
78972025-03-07T22:41:53.986ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 18.452997ms iteration = 9 status = {"errors":[],"found_port_ok":[]}
78982025-03-07T22:41:53.986ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78992025-03-07T22:41:53.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
79002025-03-07T22:41:53.988ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1272, tv_nsec: 339147538 } })
79012025-03-07T22:41:53.988ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 20.659628ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
79022025-03-07T22:41:53.988ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
79032025-03-07T22:41:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
79042025-03-07T22:41:53.988ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
79052025-03-07T22:41:53.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79062025-03-07T22:41:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79072025-03-07T22:41:53.989ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79082025-03-07T22:41:53.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79092025-03-07T22:41:53.991ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1272, tv_nsec: 342000905 } })
79102025-03-07T22:41:53.991ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79112025-03-07T22:41:53.991ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
79122025-03-07T22:41:53.992ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79132025-03-07T22:41:53.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
79142025-03-07T22:41:53.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
79152025-03-07T22:41:54.000ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79162025-03-07T22:41:54.001ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79172025-03-07T22:41:54.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
79182025-03-07T22:41:54.003ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
79192025-03-07T22:41:54.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
79202025-03-07T22:41:54.005ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
79212025-03-07T22:41:54.005ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79222025-03-07T22:41:54.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79232025-03-07T22:41:54.013ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
79242025-03-07T22:41:54.013ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
79252025-03-07T22:41:54.013ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
79262025-03-07T22:41:54.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
79272025-03-07T22:41:54.025ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 58.023527ms iteration = 17 status = {"errors":[],"requests_created_ok":[]}
79282025-03-07T22:41:54.026ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79292025-03-07T22:41:54.029ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79302025-03-07T22:41:54.029ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
79312025-03-07T22:41:54.029ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
79322025-03-07T22:41:54.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
79332025-03-07T22:41:54.030ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.596891ms iteration = 49 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
79342025-03-07T22:41:54.030ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
79352025-03-07T22:41:54.030ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 63.213538ms iteration = 49 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
79362025-03-07T22:41:54.030ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79372025-03-07T22:41:54.032ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
79382025-03-07T22:41:54.032ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
79392025-03-07T22:41:54.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
79402025-03-07T22:41:54.033ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
79412025-03-07T22:41:54.033ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
79422025-03-07T22:41:54.033ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
79432025-03-07T22:41:54.033ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 66.083085ms iteration = 17 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
79442025-03-07T22:41:54.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
79452025-03-07T22:41:54.034ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
79462025-03-07T22:41:54.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
79472025-03-07T22:41:54.035ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
79482025-03-07T22:41:54.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
79492025-03-07T22:41:54.037ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 69.051483ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
79502025-03-07T22:41:54.037ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
79512025-03-07T22:41:54.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
79522025-03-07T22:41:54.037ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79532025-03-07T22:41:54.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79542025-03-07T22:41:54.041ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
79552025-03-07T22:41:54.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
79562025-03-07T22:41:54.042ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79572025-03-07T22:41:54.043ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
79582025-03-07T22:41:54.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79592025-03-07T22:41:54.043ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 75.400154ms iteration = 17 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
79602025-03-07T22:41:54.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
79612025-03-07T22:41:54.044ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79622025-03-07T22:41:54.045ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79632025-03-07T22:41:54.046ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79642025-03-07T22:41:54.047ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
79652025-03-07T22:41:54.047ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
79662025-03-07T22:41:54.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
79672025-03-07T22:41:54.047ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
79682025-03-07T22:41:54.047ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
79692025-03-07T22:41:54.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
79702025-03-07T22:41:54.049ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
79712025-03-07T22:41:54.049ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
79722025-03-07T22:41:54.049ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
79732025-03-07T22:41:54.049ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
79742025-03-07T22:41:54.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
79752025-03-07T22:41:54.050ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
79762025-03-07T22:41:54.050ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79772025-03-07T22:41:54.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79782025-03-07T22:41:54.051ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
79792025-03-07T22:41:54.051ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
79802025-03-07T22:41:54.051ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
79812025-03-07T22:41:54.051ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
79822025-03-07T22:41:54.051ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
79832025-03-07T22:41:54.051ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
79842025-03-07T22:41:54.051ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 83.641664ms iteration = 9 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
79852025-03-07T22:41:54.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
79862025-03-07T22:41:54.053ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79872025-03-07T22:41:54.054ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
79882025-03-07T22:41:54.054ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
79892025-03-07T22:41:54.054ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
79902025-03-07T22:41:54.054ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79912025-03-07T22:41:54.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
79922025-03-07T22:41:54.054ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 86.945266ms iteration = 17 status = 0
79932025-03-07T22:41:54.058ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
79942025-03-07T22:41:54.058ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
79952025-03-07T22:41:54.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
79962025-03-07T22:41:54.058ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
79972025-03-07T22:41:54.058ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
79982025-03-07T22:41:54.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
79992025-03-07T22:41:54.086ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
80002025-03-07T22:41:54.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
80012025-03-07T22:41:54.110ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
80022025-03-07T22:41:54.110ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
80032025-03-07T22:41:54.110ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 142.361568ms iteration = 17 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
80042025-03-07T22:41:54.113ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
80052025-03-07T22:41:54.118ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
80062025-03-07T22:41:54.118ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
80072025-03-07T22:41:54.118ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
80082025-03-07T22:41:54.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
80092025-03-07T22:41:54.119ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
80102025-03-07T22:41:54.119ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 151.492416ms iteration = 17 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
80112025-03-07T22:41:54.121ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80122025-03-07T22:41:54.122ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
80132025-03-07T22:41:54.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
80142025-03-07T22:41:54.125ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
80152025-03-07T22:41:54.125ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
80162025-03-07T22:41:54.125ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
80172025-03-07T22:41:54.126ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
80182025-03-07T22:41:54.126ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
80192025-03-07T22:41:54.126ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
80202025-03-07T22:41:54.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
80212025-03-07T22:41:54.131ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
80222025-03-07T22:41:54.131ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
80232025-03-07T22:41:54.131ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-07 22:34:17.208516 UTC dns_group = internal generation = 1
80242025-03-07T22:41:54.131ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 163.286449ms iteration = 9 status = {"generation":1}
80252025-03-07T22:41:54.133ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80262025-03-07T22:41:54.133ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
80272025-03-07T22:41:54.133ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
80282025-03-07T22:41:54.134ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d1de655-926b-4d29-a0f0-ab860adeefc6", "content-length": "26", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} })
80292025-03-07T22:41:54.134ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
80302025-03-07T22:41:54.136ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
80312025-03-07T22:41:54.137ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
80322025-03-07T22:41:54.137ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
80332025-03-07T22:41:54.137ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
80342025-03-07T22:41:54.139ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
80352025-03-07T22:41:54.139ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
80362025-03-07T22:41:54.139ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-07 22:34:17.797135 UTC dns_group = external generation = 2
80372025-03-07T22:41:54.139ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 171.856461ms iteration = 9 status = {"generation":2}
80382025-03-07T22:41:54.146ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
80392025-03-07T22:41:54.146ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
80402025-03-07T22:41:54.146ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bb20277-680a-4383-8d16-bcae897e247a", "content-length": "26", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} })
80412025-03-07T22:41:54.146ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
80422025-03-07T22:41:54.149ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
80432025-03-07T22:41:54.175ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
80442025-03-07T22:41:54.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
80452025-03-07T22:41:54.177ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80462025-03-07T22:41:54.180ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
80472025-03-07T22:41:54.180ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
80482025-03-07T22:41:54.180ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
80492025-03-07T22:41:54.182ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 215.315569ms iteration = 18 status = {"physical_disks_added":0}
80502025-03-07T22:41:54.191ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80512025-03-07T22:41:54.193ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
80522025-03-07T22:41:54.193ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
80532025-03-07T22:41:54.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
80542025-03-07T22:41:54.240ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
80552025-03-07T22:41:54.244ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
80562025-03-07T22:41:54.244ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 276.587458ms iteration = 9 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
80572025-03-07T22:41:54.299ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
80582025-03-07T22:41:54.315ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
80592025-03-07T22:41:54.315ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "b612cf98-d78b-4c7b-b068-7339dfc04e5d", "content-length": "1", "date": "Fri, 07 Mar 2025 22:41:53 GMT"} })
80602025-03-07T22:41:54.318ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
80612025-03-07T22:41:54.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
80622025-03-07T22:41:54.321ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
80632025-03-07T22:41:54.321ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
80642025-03-07T22:41:54.321ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e21516d-c4e6-4849-aebe-1ea0061e0de3", "content-length": "372", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} })
80652025-03-07T22:41:54.321ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80662025-03-07T22:41:54.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80672025-03-07T22:41:54.323ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
80682025-03-07T22:41:54.323ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
80692025-03-07T22:41:54.323ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "51f53e65-1ba5-445b-91fd-6015c106181d", "content-length": "26", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} })
80702025-03-07T22:41:54.323ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
80712025-03-07T22:41:54.323ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 355.701508ms iteration = 17 status = null
80722025-03-07T22:41:54.325ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
80732025-03-07T22:41:54.353ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80742025-03-07T22:41:54.355ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
80752025-03-07T22:41:54.356ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
80762025-03-07T22:41:54.356ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
80772025-03-07T22:41:54.361ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) }
80782025-03-07T22:41:54.361ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 393.557192ms iteration = 9 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
80792025-03-07T22:41:54.368ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
80802025-03-07T22:41:54.368ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
80812025-03-07T22:41:54.368ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "0925464a-3fae-4b34-9e9c-79714212afe9", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} })
80822025-03-07T22:41:54.368ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80832025-03-07T22:41:54.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80842025-03-07T22:41:54.371ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80852025-03-07T22:41:54.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80862025-03-07T22:41:54.374ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80872025-03-07T22:41:54.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80882025-03-07T22:41:54.377ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
80892025-03-07T22:41:54.377ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
80902025-03-07T22:41:54.377ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "68a177b2-6269-4b45-b460-830b35560cda", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} })
80912025-03-07T22:41:54.379ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 411.616916ms iteration = 17 status = {}
80922025-03-07T22:41:54.440ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
80932025-03-07T22:41:54.440ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "43cb4b9c-861d-4d30-8efc-6431da8ed279", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
80942025-03-07T22:41:54.440ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
80952025-03-07T22:41:54.443ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "8e26d922-272d-4aa4-8490-672a7c0e9720", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
80962025-03-07T22:41:54.443ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
80972025-03-07T22:41:54.445ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "4ca6c65e-c0e4-44f2-8393-309e08ada444", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
80982025-03-07T22:41:54.445ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
80992025-03-07T22:41:54.448ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
81002025-03-07T22:41:54.448ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 480.798121ms iteration = 11 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
81012025-03-07T22:41:54.489ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 521.034527ms iteration = 17 status = {}
81022025-03-07T22:41:54.498ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
81032025-03-07T22:41:54.499ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef092676-5acb-4b4f-9d5e-244ba97b64b2", "content-length": "691", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} })
81042025-03-07T22:41:54.499ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
81052025-03-07T22:41:54.499ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
81062025-03-07T22:41:54.499ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
81072025-03-07T22:41:54.502ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 534.281224ms iteration = 17 status = {}
81082025-03-07T22:41:54.503ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
81092025-03-07T22:41:54.503ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "894066d6-51d2-44fa-9a0b-71379c71eda2", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
81102025-03-07T22:41:54.503ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
81112025-03-07T22:41:54.507ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "4a1418d8-fa7f-413f-bc0e-035375f5c756", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
81122025-03-07T22:41:54.507ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
81132025-03-07T22:41:54.509ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
81142025-03-07T22:41:54.509ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 541.335223ms iteration = 11 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
81152025-03-07T22:41:54.587ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81162025-03-07T22:41:54.587ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81172025-03-07T22:41:54.590ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
81182025-03-07T22:41:54.590ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81192025-03-07T22:41:54.590ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81202025-03-07T22:41:54.599ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
81212025-03-07T22:41:54.600ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "11f2a9d0-1663-44ab-94e0-e785f56004d5", "content-length": "134", "date": "Fri, 07 Mar 2025 22:41:53 GMT"} })
81222025-03-07T22:41:54.600ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
81232025-03-07T22:41:54.600ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
81242025-03-07T22:41:54.600ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
81252025-03-07T22:41:54.600ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
81262025-03-07T22:41:54.600ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81272025-03-07T22:41:54.600ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81282025-03-07T22:41:54.600ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81292025-03-07T22:41:54.600ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
81302025-03-07T22:41:54.600ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81312025-03-07T22:41:54.600ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
81322025-03-07T22:41:54.600ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
81332025-03-07T22:41:54.600ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81342025-03-07T22:41:54.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81352025-03-07T22:41:54.602ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
81362025-03-07T22:41:54.602ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "972a0d9b-271b-484c-ab67-b2b868d2e57d", "content-length": "2", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} })
81372025-03-07T22:41:54.602ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
81382025-03-07T22:41:54.603ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4f98247-2484-4a71-912a-2f8f07e8e670", "content-length": "52", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} })
81392025-03-07T22:41:54.603ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
81402025-03-07T22:41:54.603ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
81412025-03-07T22:41:54.677ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 277 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 429e8525-596a-493c-9d50-e5035da04031 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
81422025-03-07T22:41:54.790ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "7cd166ed-0294-491d-bfa8-a2ee524eabd1", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} })
81432025-03-07T22:41:54.791ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
81442025-03-07T22:41:54.791ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
81452025-03-07T22:41:54.791ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "d828ee7e-1dee-49f0-b2f3-9eb8f353408f", "content-length": "682", "date": "Fri, 07 Mar 2025 22:41:54 GMT"} })
81462025-03-07T22:41:54.791ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81472025-03-07T22:41:54.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81482025-03-07T22:41:54.795ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81492025-03-07T22:41:54.795ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81502025-03-07T22:41:54.798ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81512025-03-07T22:41:54.798ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81522025-03-07T22:41:54.801ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81532025-03-07T22:41:54.801ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81542025-03-07T22:41:54.808ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 840.517738ms iteration = 17 status = {}
81552025-03-07T22:42:03.967ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
81562025-03-07T22:42:03.967ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 50 reason = Timeout
81572025-03-07T22:42:03.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81582025-03-07T22:42:03.967ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81592025-03-07T22:42:03.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81602025-03-07T22:42:03.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81612025-03-07T22:42:03.969ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81622025-03-07T22:42:03.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81632025-03-07T22:42:03.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81642025-03-07T22:42:03.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
81652025-03-07T22:42:03.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81662025-03-07T22:42:03.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81672025-03-07T22:42:03.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81682025-03-07T22:42:03.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81692025-03-07T22:42:03.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81702025-03-07T22:42:03.978ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
81712025-03-07T22:42:03.978ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81722025-03-07T22:42:03.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81732025-03-07T22:42:03.990ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
81742025-03-07T22:42:03.990ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.562137ms iteration = 50 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
81752025-03-07T22:42:03.999ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.912787ms iteration = 50 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
81762025-03-07T22:42:04.677ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 285 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 4bdce8c5-63eb-4564-948f-c822fc31411c response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
81772025-03-07T22:42:13.967ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
81782025-03-07T22:42:13.967ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 51 reason = Timeout
81792025-03-07T22:42:13.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81802025-03-07T22:42:13.967ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81812025-03-07T22:42:13.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81822025-03-07T22:42:13.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81832025-03-07T22:42:13.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81842025-03-07T22:42:13.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
81852025-03-07T22:42:13.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81862025-03-07T22:42:13.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81872025-03-07T22:42:13.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81882025-03-07T22:42:13.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81892025-03-07T22:42:13.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81902025-03-07T22:42:13.978ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
81912025-03-07T22:42:13.978ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81922025-03-07T22:42:13.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81932025-03-07T22:42:13.988ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
81942025-03-07T22:42:13.988ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.930201ms iteration = 51 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
81952025-03-07T22:42:13.998ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81962025-03-07T22:42:13.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81972025-03-07T22:42:14.028ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 60.993786ms iteration = 51 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
81982025-03-07T22:42:14.677ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 253 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = bcf70b81-c9c6-41a0-b8f8-4be4bd548b8c response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
81992025-03-07T22:42:19.668ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:46016
82002025-03-07T22:42:19.668ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = e069e297-7e13-4559-991d-9cbc693cb089 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
82012025-03-07T22:42:19.668ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = e069e297-7e13-4559-991d-9cbc693cb089 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
82022025-03-07T22:42:19.671ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2720 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = e069e297-7e13-4559-991d-9cbc693cb089 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
82032025-03-07T22:42:19.671ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = 35a15024-1268-44ba-aac3-645aa08ff70e roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
82042025-03-07T22:42:19.671ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = 35a15024-1268-44ba-aac3-645aa08ff70e resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
82052025-03-07T22:42:19.673ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2054 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = 35a15024-1268-44ba-aac3-645aa08ff70e response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
82062025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
82072025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 52 reason = Timeout
82082025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82092025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
82102025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82112025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 18 reason = Timeout
82122025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
82132025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
82142025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
82152025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 19 reason = Timeout
82162025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
82172025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 18 reason = Timeout
82182025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
82192025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
82202025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
82212025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
82222025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
82232025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
82242025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 18 reason = Timeout
82252025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82262025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 18 reason = Timeout
82272025-03-07T22:42:23.968ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
82282025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 18 reason = Timeout
82292025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 18 reason = Timeout
82302025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
82312025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
82322025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
82332025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
82342025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
82352025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 18 reason = Timeout
82362025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
82372025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 18 reason = Timeout
82382025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
82392025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 18 reason = Timeout
82402025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
82412025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 18 reason = Timeout
82422025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
82432025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 18 reason = Timeout
82442025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
82452025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82462025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
82472025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
82482025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
82492025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
82502025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82512025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
82522025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
82532025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
82542025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
82552025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
82562025-03-07T22:42:23.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
82572025-03-07T22:42:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
82582025-03-07T22:42:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
82592025-03-07T22:42:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
82602025-03-07T22:42:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
82612025-03-07T22:42:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
82622025-03-07T22:42:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
82632025-03-07T22:42:23.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82642025-03-07T22:42:23.971ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82652025-03-07T22:42:23.972ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.950708ms iteration = 18 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
82662025-03-07T22:42:23.972ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82672025-03-07T22:42:23.972ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
82682025-03-07T22:42:23.972ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
82692025-03-07T22:42:23.972ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
82702025-03-07T22:42:23.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
82712025-03-07T22:42:23.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
82722025-03-07T22:42:23.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82732025-03-07T22:42:23.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82742025-03-07T22:42:23.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
82752025-03-07T22:42:23.974ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
82762025-03-07T22:42:23.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
82772025-03-07T22:42:23.974ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
82782025-03-07T22:42:23.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
82792025-03-07T22:42:23.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
82802025-03-07T22:42:23.974ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82812025-03-07T22:42:23.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82822025-03-07T22:42:23.975ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82832025-03-07T22:42:23.975ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1302, tv_nsec: 325856530 } })
82842025-03-07T22:42:23.975ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
82852025-03-07T22:42:23.976ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 7.830815ms iteration = 18 status = {"errors":[],"finish_invoked_ok":[]}
82862025-03-07T22:42:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
82872025-03-07T22:42:23.976ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82882025-03-07T22:42:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
82892025-03-07T22:42:23.976ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1302, tv_nsec: 327046632 } })
82902025-03-07T22:42:23.977ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
82912025-03-07T22:42:23.978ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 10.183758ms iteration = 18 status = {"errors":[],"requests_created_ok":[]}
82922025-03-07T22:42:23.978ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82932025-03-07T22:42:23.979ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82942025-03-07T22:42:23.980ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82952025-03-07T22:42:23.980ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
82962025-03-07T22:42:23.980ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
82972025-03-07T22:42:23.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
82982025-03-07T22:42:23.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
82992025-03-07T22:42:23.981ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
83002025-03-07T22:42:23.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
83012025-03-07T22:42:23.982ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
83022025-03-07T22:42:23.982ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83032025-03-07T22:42:23.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83042025-03-07T22:42:23.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83052025-03-07T22:42:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83062025-03-07T22:42:23.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
83072025-03-07T22:42:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83082025-03-07T22:42:23.984ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 16.013044ms iteration = 52 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
83092025-03-07T22:42:23.984ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83102025-03-07T22:42:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83112025-03-07T22:42:23.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1302, tv_nsec: 335330431 } })
83122025-03-07T22:42:23.985ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
83132025-03-07T22:42:23.985ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83142025-03-07T22:42:23.987ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83152025-03-07T22:42:23.987ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
83162025-03-07T22:42:23.987ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83172025-03-07T22:42:23.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83182025-03-07T22:42:23.989ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
83192025-03-07T22:42:23.989ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83202025-03-07T22:42:23.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83212025-03-07T22:42:24.011ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
83222025-03-07T22:42:24.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83232025-03-07T22:42:24.013ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
83242025-03-07T22:42:24.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
83252025-03-07T22:42:24.014ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
83262025-03-07T22:42:24.014ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
83272025-03-07T22:42:24.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83282025-03-07T22:42:24.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
83292025-03-07T22:42:24.017ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 49.279954ms iteration = 18 status = {"errors":[],"garbage_collect_requested":[]}
83302025-03-07T22:42:24.017ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
83312025-03-07T22:42:24.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
83322025-03-07T22:42:24.018ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83332025-03-07T22:42:24.020ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
83342025-03-07T22:42:24.020ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
83352025-03-07T22:42:24.020ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
83362025-03-07T22:42:24.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
83372025-03-07T22:42:24.021ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
83382025-03-07T22:42:24.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
83392025-03-07T22:42:24.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
83402025-03-07T22:42:24.021ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83412025-03-07T22:42:24.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83422025-03-07T22:42:24.022ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
83432025-03-07T22:42:24.022ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
83442025-03-07T22:42:24.022ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 54.786426ms iteration = 18 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
83452025-03-07T22:42:24.023ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
83462025-03-07T22:42:24.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
83472025-03-07T22:42:24.025ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
83482025-03-07T22:42:24.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
83492025-03-07T22:42:24.025ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
83502025-03-07T22:42:24.025ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 57.443062ms iteration = 18 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
83512025-03-07T22:42:24.026ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83522025-03-07T22:42:24.028ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
83532025-03-07T22:42:24.028ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
83542025-03-07T22:42:24.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
83552025-03-07T22:42:24.032ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
83562025-03-07T22:42:24.032ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 64.324388ms iteration = 52 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
83572025-03-07T22:42:24.039ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
83582025-03-07T22:42:24.039ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
83592025-03-07T22:42:24.039ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "565d67da-878b-457d-850c-53842f275628", "content-length": "26", "date": "Fri, 07 Mar 2025 22:42:24 GMT"} })
83602025-03-07T22:42:24.039ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
83612025-03-07T22:42:24.042ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
83622025-03-07T22:42:24.057ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
83632025-03-07T22:42:24.057ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
83642025-03-07T22:42:24.058ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "44f2bd65-6a74-4b6b-82d8-7d246fc44d3e", "content-length": "26", "date": "Fri, 07 Mar 2025 22:42:24 GMT"} })
83652025-03-07T22:42:24.058ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
83662025-03-07T22:42:24.060ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
83672025-03-07T22:42:24.060ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 92.427188ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
83682025-03-07T22:42:24.063ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83692025-03-07T22:42:24.064ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 96.224214ms iteration = 18 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
83702025-03-07T22:42:24.066ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 97.789929ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
83712025-03-07T22:42:24.068ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
83722025-03-07T22:42:24.068ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
83732025-03-07T22:42:24.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83742025-03-07T22:42:24.083ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 114.891814ms iteration = 18 status = 0
83752025-03-07T22:42:24.084ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
83762025-03-07T22:42:24.084ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
83772025-03-07T22:42:24.084ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "267e26e2-d9e8-4eef-a560-a0bdfcb07a90", "content-length": "26", "date": "Fri, 07 Mar 2025 22:42:24 GMT"} })
83782025-03-07T22:42:24.084ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
83792025-03-07T22:42:24.087ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
83802025-03-07T22:42:24.087ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
83812025-03-07T22:42:24.087ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 119.166135ms iteration = 18 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
83822025-03-07T22:42:24.128ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 160.699895ms iteration = 19 status = {"physical_disks_added":0}
83832025-03-07T22:42:24.147ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
83842025-03-07T22:42:24.147ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "5feba505-e3a7-4b63-b167-6388a599b669", "content-length": "1", "date": "Fri, 07 Mar 2025 22:42:23 GMT"} })
83852025-03-07T22:42:24.149ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
83862025-03-07T22:42:24.150ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
83872025-03-07T22:42:24.152ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 184.044368ms iteration = 18 status = null
83882025-03-07T22:42:24.162ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
83892025-03-07T22:42:24.162ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
83902025-03-07T22:42:24.162ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "62ad5f40-09c4-440d-a265-b40a3fd32c71", "content-length": "372", "date": "Fri, 07 Mar 2025 22:42:24 GMT"} })
83912025-03-07T22:42:24.162ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83922025-03-07T22:42:24.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83932025-03-07T22:42:24.163ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 195.189486ms iteration = 18 status = {}
83942025-03-07T22:42:24.167ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
83952025-03-07T22:42:24.167ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
83962025-03-07T22:42:24.169ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83972025-03-07T22:42:24.170ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
83982025-03-07T22:42:24.170ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
83992025-03-07T22:42:24.171ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
84002025-03-07T22:42:24.172ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84012025-03-07T22:42:24.174ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
84022025-03-07T22:42:24.174ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
84032025-03-07T22:42:24.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
84042025-03-07T22:42:24.176ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
84052025-03-07T22:42:24.176ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
84062025-03-07T22:42:24.176ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "70aca02c-7b47-4d9a-bf17-9dd47fbb0d71", "date": "Fri, 07 Mar 2025 22:42:24 GMT"} })
84072025-03-07T22:42:24.176ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84082025-03-07T22:42:24.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84092025-03-07T22:42:24.178ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84102025-03-07T22:42:24.179ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84112025-03-07T22:42:24.182ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84122025-03-07T22:42:24.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84132025-03-07T22:42:24.185ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
84142025-03-07T22:42:24.185ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
84152025-03-07T22:42:24.186ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "48d553e5-6098-4a26-bcb8-1bc207c5e97b", "date": "Fri, 07 Mar 2025 22:42:24 GMT"} })
84162025-03-07T22:42:24.187ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 218.807763ms iteration = 18 status = {}
84172025-03-07T22:42:24.235ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84182025-03-07T22:42:24.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84192025-03-07T22:42:24.237ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
84202025-03-07T22:42:24.237ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84212025-03-07T22:42:24.237ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84222025-03-07T22:42:24.247ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
84232025-03-07T22:42:24.247ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "932d7205-0f01-4590-80f2-d9b85f6c4379", "content-length": "134", "date": "Fri, 07 Mar 2025 22:42:23 GMT"} })
84242025-03-07T22:42:24.247ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
84252025-03-07T22:42:24.247ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
84262025-03-07T22:42:24.247ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
84272025-03-07T22:42:24.247ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
84282025-03-07T22:42:24.247ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
84292025-03-07T22:42:24.247ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
84302025-03-07T22:42:24.247ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
84312025-03-07T22:42:24.247ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
84322025-03-07T22:42:24.247ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
84332025-03-07T22:42:24.247ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
84342025-03-07T22:42:24.248ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
84352025-03-07T22:42:24.248ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84362025-03-07T22:42:24.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84372025-03-07T22:42:24.250ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
84382025-03-07T22:42:24.250ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "58dac766-e4a3-42a3-b3a9-435cd6adff8a", "content-length": "2", "date": "Fri, 07 Mar 2025 22:42:23 GMT"} })
84392025-03-07T22:42:24.250ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
84402025-03-07T22:42:24.250ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "63cbb657-aa2d-4570-ae31-87c2c816929a", "content-length": "52", "date": "Fri, 07 Mar 2025 22:42:23 GMT"} })
84412025-03-07T22:42:24.250ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
84422025-03-07T22:42:24.250ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
84432025-03-07T22:42:24.272ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
84442025-03-07T22:42:24.348ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "337a00c6-26e8-493e-9096-e0896311dec7", "date": "Fri, 07 Mar 2025 22:42:24 GMT"} })
84452025-03-07T22:42:24.348ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
84462025-03-07T22:42:24.348ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
84472025-03-07T22:42:24.348ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "0180e231-8c61-4cac-a8e6-c4d6418b8122", "content-length": "682", "date": "Fri, 07 Mar 2025 22:42:24 GMT"} })
84482025-03-07T22:42:24.348ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84492025-03-07T22:42:24.349ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84502025-03-07T22:42:24.349ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f1fc079-e485-450a-af35-07fb91f08ad8", "content-length": "691", "date": "Fri, 07 Mar 2025 22:42:24 GMT"} })
84512025-03-07T22:42:24.349ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
84522025-03-07T22:42:24.349ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
84532025-03-07T22:42:24.349ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
84542025-03-07T22:42:24.350ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 382.257163ms iteration = 18 status = {}
84552025-03-07T22:42:24.352ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84562025-03-07T22:42:24.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84572025-03-07T22:42:24.356ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84582025-03-07T22:42:24.356ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84592025-03-07T22:42:24.357ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84602025-03-07T22:42:24.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84612025-03-07T22:42:24.364ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 396.310448ms iteration = 18 status = {}
84622025-03-07T22:42:24.677ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 265 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 3c122bdb-add9-462b-995f-56cedef8b6c0 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
84632025-03-07T22:42:24.738ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
84642025-03-07T22:42:24.738ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
84652025-03-07T22:42:24.768ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
84662025-03-07T22:42:24.768ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:46722
84672025-03-07T22:42:24.768ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:46722 req_id = 49f8a995-5f91-41c2-a21f-3a1a25d0ec94 roles = RoleSet { roles: {} } uri = /metrics/producers
84682025-03-07T22:42:24.768ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:46722 req_id = 49f8a995-5f91-41c2-a21f-3a1a25d0ec94 resource = Database result = Ok(()) uri = /metrics/producers
84692025-03-07T22:42:24.808ZINFOnexus: assigned collector to new producer collector_id = 7b5e9a24-6b39-4a58-a275-8cc5a75195d8 file = nexus/src/app/oximeter.rs:76 producer_id = dfe169a3-7536-4b54-9593-8815d6e81dd4
84702025-03-07T22:42:24.808ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 39998 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:46722 req_id = 49f8a995-5f91-41c2-a21f-3a1a25d0ec94 response_code = 201 uri = /metrics/producers
84712025-03-07T22:42:24.808ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "49f8a995-5f91-41c2-a21f-3a1a25d0ec94", "content-length": "41", "date": "Fri, 07 Mar 2025 22:42:24 GMT"} })
84722025-03-07T22:42:24.810ZDEBGnexus (producer-registration-task): registered with nexus successfully
84732025-03-07T22:42:24.810ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
84742025-03-07T22:42:33.967ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
84752025-03-07T22:42:33.967ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 53 reason = Timeout
84762025-03-07T22:42:33.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84772025-03-07T22:42:33.967ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
84782025-03-07T22:42:33.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84792025-03-07T22:42:33.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
84802025-03-07T22:42:33.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84812025-03-07T22:42:33.970ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
84822025-03-07T22:42:33.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
84832025-03-07T22:42:33.972ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.944588ms iteration = 53 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
84842025-03-07T22:42:33.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
84852025-03-07T22:42:33.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84862025-03-07T22:42:33.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84872025-03-07T22:42:33.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84882025-03-07T22:42:33.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84892025-03-07T22:42:33.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84902025-03-07T22:42:33.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
84912025-03-07T22:42:33.979ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84922025-03-07T22:42:33.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84932025-03-07T22:42:33.989ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
84942025-03-07T22:42:33.989ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.545467ms iteration = 53 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
84952025-03-07T22:42:34.668ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = 14c140bc-b822-452c-b132-fd465e28f332 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
84962025-03-07T22:42:34.669ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = 14c140bc-b822-452c-b132-fd465e28f332 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
84972025-03-07T22:42:34.672ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3535 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = 14c140bc-b822-452c-b132-fd465e28f332 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
84982025-03-07T22:42:34.672ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = ce29676a-e49f-4d8c-b223-7cf9aeb9ac77 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
84992025-03-07T22:42:34.672ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = ce29676a-e49f-4d8c-b223-7cf9aeb9ac77 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
85002025-03-07T22:42:34.674ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2463 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = ce29676a-e49f-4d8c-b223-7cf9aeb9ac77 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
85012025-03-07T22:42:34.677ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 229 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 90bdf905-b098-4100-aa79-e9dbf509ff3e response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
85022025-03-07T22:42:43.968ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 54 reason = Timeout
85032025-03-07T22:42:43.968ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 54 reason = Timeout
85042025-03-07T22:42:43.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85052025-03-07T22:42:43.968ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85062025-03-07T22:42:43.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85072025-03-07T22:42:43.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85082025-03-07T22:42:43.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85092025-03-07T22:42:43.972ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85102025-03-07T22:42:43.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85112025-03-07T22:42:43.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
85122025-03-07T22:42:43.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85132025-03-07T22:42:43.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85142025-03-07T22:42:43.974ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 6.36509ms iteration = 54 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
85152025-03-07T22:42:43.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85162025-03-07T22:42:43.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85172025-03-07T22:42:43.979ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85182025-03-07T22:42:43.982ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
85192025-03-07T22:42:43.982ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85202025-03-07T22:42:43.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85212025-03-07T22:42:43.997ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
85222025-03-07T22:42:43.997ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 29.400693ms iteration = 54 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
85232025-03-07T22:42:44.678ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 332 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = e2652665-f4fe-4f6a-b186-20e546d2220f response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
85242025-03-07T22:42:53.967ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
85252025-03-07T22:42:53.967ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 19 reason = Timeout
85262025-03-07T22:42:53.967ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
85272025-03-07T22:42:53.967ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 19 reason = Timeout
85282025-03-07T22:42:53.967ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
85292025-03-07T22:42:53.967ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
85302025-03-07T22:42:53.967ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 19 reason = Timeout
85312025-03-07T22:42:53.967ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 10 reason = Timeout
85322025-03-07T22:42:53.967ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
85332025-03-07T22:42:53.967ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
85342025-03-07T22:42:53.967ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
85352025-03-07T22:42:53.967ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 19 reason = Timeout
85362025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 19 reason = Timeout
85372025-03-07T22:42:53.968ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
85382025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
85392025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
85402025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 20 reason = Timeout
85412025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
85422025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 10 reason = Timeout
85432025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 19 reason = Timeout
85442025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
85452025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 5 reason = Timeout
85462025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85472025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
85482025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
85492025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 55 reason = Timeout
85502025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85512025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 19 reason = Timeout
85522025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
85532025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
85542025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
85552025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
85562025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 11 reason = Timeout
85572025-03-07T22:42:53.968ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
85582025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
85592025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 19.74\u{b5}s iteration = 11 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
85602025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
85612025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
85622025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
85632025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
85642025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 19 reason = Timeout
85652025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
85662025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
85672025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
85682025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 19 reason = Timeout
85692025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 55 reason = Timeout
85702025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
85712025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85722025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 19 reason = Timeout
85732025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 10 reason = Timeout
85742025-03-07T22:42:53.968ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:32:53.968310035 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
85752025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
85762025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 19 reason = Timeout
85772025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
85782025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
85792025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
85802025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
85812025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
85822025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
85832025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
85842025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
85852025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85862025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
85872025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
85882025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
85892025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
85902025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
85912025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85922025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 19 reason = Timeout
85932025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
85942025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 19 reason = Timeout
85952025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
85962025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
85972025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 10 reason = Timeout
85982025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
85992025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
86002025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 19 reason = Timeout
86012025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
86022025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 19 reason = Timeout
86032025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
86042025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
86052025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 10 reason = Timeout
86062025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
86072025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
86082025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
86092025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
86102025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 19 reason = Timeout
86112025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
86122025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
86132025-03-07T22:42:53.968ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1332, tv_nsec: 318527200 } })
86142025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
86152025-03-07T22:42:53.968ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
86162025-03-07T22:42:53.968ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1332, tv_nsec: 318584631 } })
86172025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
86182025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
86192025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
86202025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
86212025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
86222025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
86232025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
86242025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
86252025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
86262025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
86272025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.10391ms iteration = 10 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
86282025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
86292025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
86302025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
86312025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.291752ms iteration = 10 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
86322025-03-07T22:42:53.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
86332025-03-07T22:42:53.970ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86342025-03-07T22:42:53.971ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86352025-03-07T22:42:53.971ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86362025-03-07T22:42:53.971ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.647006ms iteration = 19 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
86372025-03-07T22:42:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
86382025-03-07T22:42:53.972ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
86392025-03-07T22:42:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
86402025-03-07T22:42:53.973ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
86412025-03-07T22:42:53.973ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
86422025-03-07T22:42:53.973ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
86432025-03-07T22:42:53.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
86442025-03-07T22:42:53.973ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
86452025-03-07T22:42:53.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
86462025-03-07T22:42:53.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
86472025-03-07T22:42:53.973ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
86482025-03-07T22:42:53.973ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
86492025-03-07T22:42:53.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
86502025-03-07T22:42:53.975ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86512025-03-07T22:42:53.975ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86522025-03-07T22:42:53.975ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86532025-03-07T22:42:53.976ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86542025-03-07T22:42:53.976ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86552025-03-07T22:42:53.978ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
86562025-03-07T22:42:53.978ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
86572025-03-07T22:42:53.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
86582025-03-07T22:42:53.978ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
86592025-03-07T22:42:53.978ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86602025-03-07T22:42:53.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86612025-03-07T22:42:53.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
86622025-03-07T22:42:53.979ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
86632025-03-07T22:42:53.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
86642025-03-07T22:42:53.979ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
86652025-03-07T22:42:53.979ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
86662025-03-07T22:42:53.979ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
86672025-03-07T22:42:53.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
86682025-03-07T22:42:53.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
86692025-03-07T22:42:53.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
86702025-03-07T22:42:53.980ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86712025-03-07T22:42:53.980ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
86722025-03-07T22:42:53.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
86732025-03-07T22:42:53.981ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86742025-03-07T22:42:53.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
86752025-03-07T22:42:53.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
86762025-03-07T22:42:53.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
86772025-03-07T22:42:53.982ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
86782025-03-07T22:42:53.982ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86792025-03-07T22:42:53.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
86802025-03-07T22:42:53.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
86812025-03-07T22:42:53.983ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
86822025-03-07T22:42:53.983ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1332, tv_nsec: 333050610 } })
86832025-03-07T22:42:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
86842025-03-07T22:42:53.983ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
86852025-03-07T22:42:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
86862025-03-07T22:42:53.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
86872025-03-07T22:42:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
86882025-03-07T22:42:53.984ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1332, tv_nsec: 334285881 } })
86892025-03-07T22:42:53.984ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
86902025-03-07T22:42:53.985ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:32:53.968310035 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
86912025-03-07T22:42:53.985ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 16.954102ms iteration = 10 status = {"expiration":"2025-03-07T22:32:53.968310035Z","pruned":[]}
86922025-03-07T22:42:53.985ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86932025-03-07T22:42:53.986ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
86942025-03-07T22:42:53.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
86952025-03-07T22:42:53.986ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 17.960053ms iteration = 10 status = {"errors":[],"found_port_ok":[]}
86962025-03-07T22:42:53.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
86972025-03-07T22:42:53.987ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
86982025-03-07T22:42:53.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
86992025-03-07T22:42:53.988ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
87002025-03-07T22:42:53.988ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
87012025-03-07T22:42:53.988ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 19.380727ms iteration = 10 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
87022025-03-07T22:42:53.989ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87032025-03-07T22:42:53.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
87042025-03-07T22:42:53.991ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
87052025-03-07T22:42:53.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
87062025-03-07T22:42:53.991ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87072025-03-07T22:42:53.992ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
87082025-03-07T22:42:53.992ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
87092025-03-07T22:42:53.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
87102025-03-07T22:42:53.992ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87112025-03-07T22:42:53.994ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87122025-03-07T22:42:53.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
87132025-03-07T22:42:53.994ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
87142025-03-07T22:42:53.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
87152025-03-07T22:42:53.995ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
87162025-03-07T22:42:53.995ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
87172025-03-07T22:42:53.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
87182025-03-07T22:42:53.998ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
87192025-03-07T22:42:53.998ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
87202025-03-07T22:42:53.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
87212025-03-07T22:42:53.998ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1332, tv_nsec: 348075584 } })
87222025-03-07T22:42:53.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
87232025-03-07T22:42:53.998ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
87242025-03-07T22:42:53.998ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87252025-03-07T22:42:53.998ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87262025-03-07T22:42:53.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87272025-03-07T22:42:54.000ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.965088ms iteration = 55 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
87282025-03-07T22:42:54.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
87292025-03-07T22:42:54.006ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87302025-03-07T22:42:54.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87312025-03-07T22:42:54.008ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87322025-03-07T22:42:54.011ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
87332025-03-07T22:42:54.011ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
87342025-03-07T22:42:54.011ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
87352025-03-07T22:42:54.011ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
87362025-03-07T22:42:54.011ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
87372025-03-07T22:42:54.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
87382025-03-07T22:42:54.012ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 44.333906ms iteration = 5 status = {"nsuccess":5}
87392025-03-07T22:42:54.014ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87402025-03-07T22:42:54.017ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
87412025-03-07T22:42:54.017ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
87422025-03-07T22:42:54.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
87432025-03-07T22:42:54.019ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87442025-03-07T22:42:54.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
87452025-03-07T22:42:54.021ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
87462025-03-07T22:42:54.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
87472025-03-07T22:42:54.023ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87482025-03-07T22:42:54.026ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
87492025-03-07T22:42:54.026ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
87502025-03-07T22:42:54.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
87512025-03-07T22:42:54.027ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
87522025-03-07T22:42:54.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
87532025-03-07T22:42:54.028ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 59.31997ms iteration = 19 status = {"errors":[],"garbage_collect_requested":[]}
87542025-03-07T22:42:54.029ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 61.083447ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
87552025-03-07T22:42:54.030ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 62.55558ms iteration = 20 status = {"physical_disks_added":0}
87562025-03-07T22:42:54.041ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
87572025-03-07T22:42:54.041ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 73.552456ms iteration = 19 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
87582025-03-07T22:42:54.044ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 75.758908ms iteration = 19 status = {"errors":[],"requests_created_ok":[]}
87592025-03-07T22:42:54.047ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 78.422903ms iteration = 19 status = {"errors":[],"finish_invoked_ok":[]}
87602025-03-07T22:42:54.048ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
87612025-03-07T22:42:54.048ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
87622025-03-07T22:42:54.048ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 80.482883ms iteration = 10 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
87632025-03-07T22:42:54.049ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
87642025-03-07T22:42:54.049ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
87652025-03-07T22:42:54.049ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
87662025-03-07T22:42:54.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
87672025-03-07T22:42:54.054ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
87682025-03-07T22:42:54.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
87692025-03-07T22:42:54.058ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
87702025-03-07T22:42:54.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
87712025-03-07T22:42:54.059ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 91.058994ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
87722025-03-07T22:42:54.060ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87732025-03-07T22:42:54.061ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
87742025-03-07T22:42:54.061ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
87752025-03-07T22:42:54.061ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
87762025-03-07T22:42:54.061ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
87772025-03-07T22:42:54.062ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
87782025-03-07T22:42:54.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
87792025-03-07T22:42:54.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
87802025-03-07T22:42:54.063ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
87812025-03-07T22:42:54.063ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
87822025-03-07T22:42:54.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
87832025-03-07T22:42:54.063ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87842025-03-07T22:42:54.065ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
87852025-03-07T22:42:54.065ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 96.884811ms iteration = 19 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
87862025-03-07T22:42:54.067ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
87872025-03-07T22:42:54.067ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
87882025-03-07T22:42:54.067ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
87892025-03-07T22:42:54.067ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
87902025-03-07T22:42:54.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
87912025-03-07T22:42:54.068ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87922025-03-07T22:42:54.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87932025-03-07T22:42:54.069ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87942025-03-07T22:42:54.071ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87952025-03-07T22:42:54.072ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87962025-03-07T22:42:54.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87972025-03-07T22:42:54.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
87982025-03-07T22:42:54.072ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
87992025-03-07T22:42:54.072ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
88002025-03-07T22:42:54.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
88012025-03-07T22:42:54.074ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88022025-03-07T22:42:54.075ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88032025-03-07T22:42:54.076ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
88042025-03-07T22:42:54.078ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
88052025-03-07T22:42:54.078ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88062025-03-07T22:42:54.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88072025-03-07T22:42:54.078ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
88082025-03-07T22:42:54.078ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88092025-03-07T22:42:54.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88102025-03-07T22:42:54.081ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 113.727482ms iteration = 19 status = 0
88112025-03-07T22:42:54.082ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) }
88122025-03-07T22:42:54.082ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 114.210087ms iteration = 10 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
88132025-03-07T22:42:54.085ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
88142025-03-07T22:42:54.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
88152025-03-07T22:42:54.087ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88162025-03-07T22:42:54.089ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
88172025-03-07T22:42:54.089ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
88182025-03-07T22:42:54.089ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
88192025-03-07T22:42:54.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
88202025-03-07T22:42:54.090ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
88212025-03-07T22:42:54.090ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
88222025-03-07T22:42:54.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
88232025-03-07T22:42:54.092ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88242025-03-07T22:42:54.092ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88252025-03-07T22:42:54.095ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
88262025-03-07T22:42:54.095ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
88272025-03-07T22:42:54.096ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
88282025-03-07T22:42:54.096ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
88292025-03-07T22:42:54.096ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 127.99881ms iteration = 55 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
88302025-03-07T22:42:54.096ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
88312025-03-07T22:42:54.096ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
88322025-03-07T22:42:54.097ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
88332025-03-07T22:42:54.109ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
88342025-03-07T22:42:54.109ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
88352025-03-07T22:42:54.109ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-07 22:34:17.797135 UTC dns_group = external generation = 2
88362025-03-07T22:42:54.109ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 140.824013ms iteration = 10 status = {"generation":2}
88372025-03-07T22:42:54.116ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
88382025-03-07T22:42:54.116ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
88392025-03-07T22:42:54.116ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
88402025-03-07T22:42:54.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
88412025-03-07T22:42:54.117ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 148.424646ms iteration = 19 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
88422025-03-07T22:42:54.117ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
88432025-03-07T22:42:54.117ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
88442025-03-07T22:42:54.117ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 150.030532ms iteration = 19 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
88452025-03-07T22:42:54.137ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
88462025-03-07T22:42:54.137ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
88472025-03-07T22:42:54.138ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "04d1d1e7-53ab-476c-9e75-caca7859b6cd", "content-length": "26", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} })
88482025-03-07T22:42:54.138ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
88492025-03-07T22:42:54.140ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
88502025-03-07T22:42:54.169ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
88512025-03-07T22:42:54.169ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
88522025-03-07T22:42:54.169ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "43cb1e15-2bcf-418d-8f76-b4205ed7f4c4", "content-length": "26", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} })
88532025-03-07T22:42:54.169ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
88542025-03-07T22:42:54.171ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
88552025-03-07T22:42:54.198ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
88562025-03-07T22:42:54.215ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
88572025-03-07T22:42:54.215ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 247.881171ms iteration = 10 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
88582025-03-07T22:42:54.290ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
88592025-03-07T22:42:54.291ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf23ca3a-a23e-431c-86e2-23912e355a84", "content-length": "691", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} })
88602025-03-07T22:42:54.291ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
88612025-03-07T22:42:54.291ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
88622025-03-07T22:42:54.291ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
88632025-03-07T22:42:54.292ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 324.612639ms iteration = 19 status = {}
88642025-03-07T22:42:54.292ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88652025-03-07T22:42:54.295ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
88662025-03-07T22:42:54.295ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
88672025-03-07T22:42:54.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
88682025-03-07T22:42:54.302ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
88692025-03-07T22:42:54.302ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
88702025-03-07T22:42:54.302ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8be313a3-b4e1-4f7d-b3c0-46d4647e044c", "content-length": "26", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} })
88712025-03-07T22:42:54.302ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
88722025-03-07T22:42:54.304ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
88732025-03-07T22:42:54.398ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
88742025-03-07T22:42:54.399ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "dce3952f-f0fd-4221-bdb5-4172d53654e4", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
88752025-03-07T22:42:54.399ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
88762025-03-07T22:42:54.400ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "2340750d-ee68-4081-85bc-2ab27a30567e", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
88772025-03-07T22:42:54.400ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
88782025-03-07T22:42:54.402ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
88792025-03-07T22:42:54.402ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 434.886969ms iteration = 12 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
88802025-03-07T22:42:54.422ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
88812025-03-07T22:42:54.422ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
88822025-03-07T22:42:54.422ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "a20a5ee5-7068-4745-9a00-fd2550ae6a01", "content-length": "372", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} })
88832025-03-07T22:42:54.422ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88842025-03-07T22:42:54.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88852025-03-07T22:42:54.426ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
88862025-03-07T22:42:54.426ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
88872025-03-07T22:42:54.426ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-07 22:34:17.208516 UTC dns_group = internal generation = 1
88882025-03-07T22:42:54.426ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 457.91375ms iteration = 10 status = {"generation":1}
88892025-03-07T22:42:54.427ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
88902025-03-07T22:42:54.427ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 459.236943ms iteration = 19 status = {}
88912025-03-07T22:42:54.458ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
88922025-03-07T22:42:54.458ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f04ca1a-01fa-4eef-89c7-0f5297ba2fb6", "content-length": "1", "date": "Fri, 07 Mar 2025 22:42:53 GMT"} })
88932025-03-07T22:42:54.461ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
88942025-03-07T22:42:54.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
88952025-03-07T22:42:54.474ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 506.117363ms iteration = 19 status = null
88962025-03-07T22:42:54.511ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
88972025-03-07T22:42:54.511ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
88982025-03-07T22:42:54.512ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "6b8b05e5-7dce-4931-afa7-9d522d4c8467", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} })
88992025-03-07T22:42:54.512ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89002025-03-07T22:42:54.512ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89012025-03-07T22:42:54.515ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89022025-03-07T22:42:54.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89032025-03-07T22:42:54.520ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89042025-03-07T22:42:54.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89052025-03-07T22:42:54.525ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
89062025-03-07T22:42:54.525ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
89072025-03-07T22:42:54.526ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "0c8b5580-513a-4ecf-949f-8bc792637a78", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} })
89082025-03-07T22:42:54.528ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 560.022562ms iteration = 19 status = {}
89092025-03-07T22:42:54.537ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
89102025-03-07T22:42:54.538ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "4811feef-ecf4-48ab-9036-e72155e24418", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89112025-03-07T22:42:54.538ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89122025-03-07T22:42:54.541ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "99284ded-9cf3-4069-9189-14136b81b72b", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89132025-03-07T22:42:54.541ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89142025-03-07T22:42:54.542ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "80183612-044f-4a84-a94c-7bf3ed37f119", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89152025-03-07T22:42:54.542ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89162025-03-07T22:42:54.545ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89172025-03-07T22:42:54.545ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 577.253667ms iteration = 12 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
89182025-03-07T22:42:54.580ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89192025-03-07T22:42:54.581ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89202025-03-07T22:42:54.582ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
89212025-03-07T22:42:54.583ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89222025-03-07T22:42:54.583ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89232025-03-07T22:42:54.594ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
89242025-03-07T22:42:54.594ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "f453dd19-1f10-4067-a1a7-9a2063192a7d", "content-length": "134", "date": "Fri, 07 Mar 2025 22:42:53 GMT"} })
89252025-03-07T22:42:54.595ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
89262025-03-07T22:42:54.595ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
89272025-03-07T22:42:54.595ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
89282025-03-07T22:42:54.595ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
89292025-03-07T22:42:54.595ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89302025-03-07T22:42:54.595ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89312025-03-07T22:42:54.595ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89322025-03-07T22:42:54.595ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
89332025-03-07T22:42:54.595ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89342025-03-07T22:42:54.595ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
89352025-03-07T22:42:54.595ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
89362025-03-07T22:42:54.596ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89372025-03-07T22:42:54.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89382025-03-07T22:42:54.600ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
89392025-03-07T22:42:54.601ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8199b08-0890-4745-be74-6f1565ef0f08", "content-length": "2", "date": "Fri, 07 Mar 2025 22:42:53 GMT"} })
89402025-03-07T22:42:54.601ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
89412025-03-07T22:42:54.601ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae37bd47-99f6-4095-9a5e-d06e185e140f", "content-length": "52", "date": "Fri, 07 Mar 2025 22:42:53 GMT"} })
89422025-03-07T22:42:54.601ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
89432025-03-07T22:42:54.601ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
89442025-03-07T22:42:54.678ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 306 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 57261c9f-3938-4dac-ab62-0f1a33a7afc5 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
89452025-03-07T22:42:54.721ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "7ba21609-ffd0-4a00-aa70-63435dbc6322", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} })
89462025-03-07T22:42:54.721ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
89472025-03-07T22:42:54.721ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
89482025-03-07T22:42:54.722ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6a05f34-b969-4ad6-9bed-71df2025a487", "content-length": "682", "date": "Fri, 07 Mar 2025 22:42:54 GMT"} })
89492025-03-07T22:42:54.722ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89502025-03-07T22:42:54.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89512025-03-07T22:42:54.726ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89522025-03-07T22:42:54.727ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89532025-03-07T22:42:54.731ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89542025-03-07T22:42:54.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89552025-03-07T22:42:54.734ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89562025-03-07T22:42:54.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89572025-03-07T22:42:54.741ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 773.001758ms iteration = 19 status = {}
89582025-03-07T22:43:03.968ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
89592025-03-07T22:43:03.969ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 56 reason = Timeout
89602025-03-07T22:43:03.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89612025-03-07T22:43:03.969ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
89622025-03-07T22:43:03.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89632025-03-07T22:43:03.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
89642025-03-07T22:43:03.999ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89652025-03-07T22:43:04.001ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
89662025-03-07T22:43:04.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
89672025-03-07T22:43:04.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
89682025-03-07T22:43:04.001ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89692025-03-07T22:43:04.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89702025-03-07T22:43:04.003ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89712025-03-07T22:43:04.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89722025-03-07T22:43:04.005ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89732025-03-07T22:43:04.007ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
89742025-03-07T22:43:04.007ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89752025-03-07T22:43:04.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89762025-03-07T22:43:04.019ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
89772025-03-07T22:43:04.020ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 51.032361ms iteration = 56 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
89782025-03-07T22:43:04.031ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.541701ms iteration = 56 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
89792025-03-07T22:43:04.669ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = 4f607f2c-4cbb-401e-be0d-022f88ee0d31 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
89802025-03-07T22:43:04.669ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = 4f607f2c-4cbb-401e-be0d-022f88ee0d31 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
89812025-03-07T22:43:04.671ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2409 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = 4f607f2c-4cbb-401e-be0d-022f88ee0d31 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
89822025-03-07T22:43:04.671ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = 0b208917-c800-4f3e-b05b-dca55a769937 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
89832025-03-07T22:43:04.672ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = 0b208917-c800-4f3e-b05b-dca55a769937 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
89842025-03-07T22:43:04.674ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2272 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46016 req_id = 0b208917-c800-4f3e-b05b-dca55a769937 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
89852025-03-07T22:43:04.678ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 287 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 46d0ba6b-5c66-4ac3-a08b-55e4b0b1c64a response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
89862025-03-07T22:43:13.969ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 57 reason = Timeout
89872025-03-07T22:43:13.969ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 57 reason = Timeout
89882025-03-07T22:43:13.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89892025-03-07T22:43:13.969ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
89902025-03-07T22:43:13.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
89912025-03-07T22:43:13.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89922025-03-07T22:43:13.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89932025-03-07T22:43:13.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
89942025-03-07T22:43:13.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89952025-03-07T22:43:13.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89962025-03-07T22:43:13.977ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89972025-03-07T22:43:13.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89982025-03-07T22:43:13.979ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89992025-03-07T22:43:13.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
90002025-03-07T22:43:13.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90012025-03-07T22:43:13.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90022025-03-07T22:43:13.994ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
90032025-03-07T22:43:13.994ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.749747ms iteration = 57 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
90042025-03-07T22:43:13.997ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90052025-03-07T22:43:13.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90062025-03-07T22:43:14.026ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 56.944987ms iteration = 57 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
90072025-03-07T22:43:14.678ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 293 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = de123aa4-f0c3-4ac2-8ec2-733a717c191d response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
90082025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
90092025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
90102025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 20 reason = Timeout
90112025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
90122025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
90132025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
90142025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 20 reason = Timeout
90152025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
90162025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
90172025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
90182025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 58 reason = Timeout
90192025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
90202025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90212025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 20 reason = Timeout
90222025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
90232025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 20 reason = Timeout
90242025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
90252025-03-07T22:43:23.968ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
90262025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
90272025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
90282025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 20 reason = Timeout
90292025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
90302025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90312025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 58 reason = Timeout
90322025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90332025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 21 reason = Timeout
90342025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 20 reason = Timeout
90352025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
90362025-03-07T22:43:23.968ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90372025-03-07T22:43:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
90382025-03-07T22:43:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
90392025-03-07T22:43:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
90402025-03-07T22:43:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
90412025-03-07T22:43:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
90422025-03-07T22:43:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
90432025-03-07T22:43:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90442025-03-07T22:43:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
90452025-03-07T22:43:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
90462025-03-07T22:43:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90472025-03-07T22:43:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90482025-03-07T22:43:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90492025-03-07T22:43:23.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
90502025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 20 reason = Timeout
90512025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
90522025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 20 reason = Timeout
90532025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
90542025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 20 reason = Timeout
90552025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
90562025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 20 reason = Timeout
90572025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
90582025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 20 reason = Timeout
90592025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
90602025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
90612025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
90622025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
90632025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
90642025-03-07T22:43:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
90652025-03-07T22:43:23.972ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1362, tv_nsec: 321391447 } })
90662025-03-07T22:43:23.972ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1362, tv_nsec: 321554209 } })
90672025-03-07T22:43:23.972ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.947468ms iteration = 20 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
90682025-03-07T22:43:23.972ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
90692025-03-07T22:43:23.972ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
90702025-03-07T22:43:23.974ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90712025-03-07T22:43:23.974ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90722025-03-07T22:43:23.976ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
90732025-03-07T22:43:23.976ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
90742025-03-07T22:43:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
90752025-03-07T22:43:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
90762025-03-07T22:43:23.976ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
90772025-03-07T22:43:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
90782025-03-07T22:43:23.979ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90792025-03-07T22:43:23.981ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
90802025-03-07T22:43:23.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
90812025-03-07T22:43:23.981ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
90822025-03-07T22:43:23.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
90832025-03-07T22:43:23.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
90842025-03-07T22:43:23.983ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 12.813443ms iteration = 20 status = {"errors":[],"finish_invoked_ok":[]}
90852025-03-07T22:43:23.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90862025-03-07T22:43:23.985ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
90872025-03-07T22:43:23.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
90882025-03-07T22:43:23.985ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
90892025-03-07T22:43:23.985ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
90902025-03-07T22:43:23.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
90912025-03-07T22:43:23.986ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90922025-03-07T22:43:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
90932025-03-07T22:43:23.988ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
90942025-03-07T22:43:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
90952025-03-07T22:43:24.009ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90962025-03-07T22:43:24.016ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
90972025-03-07T22:43:24.016ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
90982025-03-07T22:43:24.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
90992025-03-07T22:43:24.023ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91002025-03-07T22:43:24.023ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
91012025-03-07T22:43:24.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
91022025-03-07T22:43:24.025ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91032025-03-07T22:43:24.025ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
91042025-03-07T22:43:24.025ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91052025-03-07T22:43:24.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91062025-03-07T22:43:24.027ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
91072025-03-07T22:43:24.027ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
91082025-03-07T22:43:24.027ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
91092025-03-07T22:43:24.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
91102025-03-07T22:43:24.027ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91112025-03-07T22:43:24.027ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.201749ms iteration = 58 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
91122025-03-07T22:43:24.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91132025-03-07T22:43:24.028ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
91142025-03-07T22:43:24.028ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
91152025-03-07T22:43:24.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
91162025-03-07T22:43:24.029ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 59.673593ms iteration = 20 status = {"errors":[],"garbage_collect_requested":[]}
91172025-03-07T22:43:24.030ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
91182025-03-07T22:43:24.030ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91192025-03-07T22:43:24.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
91202025-03-07T22:43:24.031ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 61.203738ms iteration = 20 status = {"errors":[],"requests_created_ok":[]}
91212025-03-07T22:43:24.033ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
91222025-03-07T22:43:24.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
91232025-03-07T22:43:24.033ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
91242025-03-07T22:43:24.033ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
91252025-03-07T22:43:24.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
91262025-03-07T22:43:24.034ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
91272025-03-07T22:43:24.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
91282025-03-07T22:43:24.036ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 65.915113ms iteration = 20 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
91292025-03-07T22:43:24.037ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1362, tv_nsec: 385990478 } })
91302025-03-07T22:43:24.037ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
91312025-03-07T22:43:24.038ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
91322025-03-07T22:43:24.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
91332025-03-07T22:43:24.040ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
91342025-03-07T22:43:24.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
91352025-03-07T22:43:24.042ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91362025-03-07T22:43:24.044ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
91372025-03-07T22:43:24.044ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
91382025-03-07T22:43:24.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
91392025-03-07T22:43:24.057ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
91402025-03-07T22:43:24.057ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 88.614601ms iteration = 20 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
91412025-03-07T22:43:24.060ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
91422025-03-07T22:43:24.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
91432025-03-07T22:43:24.063ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
91442025-03-07T22:43:24.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
91452025-03-07T22:43:24.065ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91462025-03-07T22:43:24.069ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
91472025-03-07T22:43:24.069ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
91482025-03-07T22:43:24.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
91492025-03-07T22:43:24.071ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 102.576395ms iteration = 20 status = 0
91502025-03-07T22:43:24.072ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 104.140921ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
91512025-03-07T22:43:24.073ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
91522025-03-07T22:43:24.073ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
91532025-03-07T22:43:24.073ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
91542025-03-07T22:43:24.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
91552025-03-07T22:43:24.074ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91562025-03-07T22:43:24.078ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
91572025-03-07T22:43:24.078ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91582025-03-07T22:43:24.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91592025-03-07T22:43:24.089ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 119.482018ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
91602025-03-07T22:43:24.098ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
91612025-03-07T22:43:24.098ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 129.811307ms iteration = 20 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
91622025-03-07T22:43:24.100ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
91632025-03-07T22:43:24.100ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 132.259651ms iteration = 58 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
91642025-03-07T22:43:24.108ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
91652025-03-07T22:43:24.108ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
91662025-03-07T22:43:24.108ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 139.992545ms iteration = 20 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
91672025-03-07T22:43:24.121ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
91682025-03-07T22:43:24.122ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
91692025-03-07T22:43:24.122ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 153.573096ms iteration = 21 status = {"physical_disks_added":0}
91702025-03-07T22:43:24.158ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
91712025-03-07T22:43:24.158ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
91722025-03-07T22:43:24.159ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdbb910a-2299-42cc-b571-fd367dffad85", "content-length": "26", "date": "Fri, 07 Mar 2025 22:43:24 GMT"} })
91732025-03-07T22:43:24.159ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
91742025-03-07T22:43:24.161ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
91752025-03-07T22:43:24.179ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
91762025-03-07T22:43:24.179ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
91772025-03-07T22:43:24.180ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "012b325d-b166-44cf-927a-4dd5d0d3313e", "content-length": "26", "date": "Fri, 07 Mar 2025 22:43:24 GMT"} })
91782025-03-07T22:43:24.180ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
91792025-03-07T22:43:24.180ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
91802025-03-07T22:43:24.180ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
91812025-03-07T22:43:24.180ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d83ab4e-7658-471d-9184-220640d90a27", "content-length": "26", "date": "Fri, 07 Mar 2025 22:43:24 GMT"} })
91822025-03-07T22:43:24.180ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
91832025-03-07T22:43:24.184ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
91842025-03-07T22:43:24.184ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
91852025-03-07T22:43:24.184ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
91862025-03-07T22:43:24.184ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
91872025-03-07T22:43:24.185ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "be73b0b5-47a5-4918-8948-f09f797674d1", "content-length": "372", "date": "Fri, 07 Mar 2025 22:43:24 GMT"} })
91882025-03-07T22:43:24.185ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91892025-03-07T22:43:24.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91902025-03-07T22:43:24.236ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
91912025-03-07T22:43:24.236ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
91922025-03-07T22:43:24.236ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "11b23e92-ab7f-40d7-bdc8-5e685aa2919e", "date": "Fri, 07 Mar 2025 22:43:24 GMT"} })
91932025-03-07T22:43:24.236ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91942025-03-07T22:43:24.237ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91952025-03-07T22:43:24.239ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91962025-03-07T22:43:24.239ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91972025-03-07T22:43:24.242ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91982025-03-07T22:43:24.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91992025-03-07T22:43:24.246ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
92002025-03-07T22:43:24.246ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
92012025-03-07T22:43:24.247ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "6aa8f96f-e165-4f44-b415-de95f7d1af0e", "date": "Fri, 07 Mar 2025 22:43:24 GMT"} })
92022025-03-07T22:43:24.248ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 280.01478ms iteration = 20 status = {}
92032025-03-07T22:43:24.295ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
92042025-03-07T22:43:24.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
92052025-03-07T22:43:24.298ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92062025-03-07T22:43:24.300ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
92072025-03-07T22:43:24.300ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
92082025-03-07T22:43:24.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
92092025-03-07T22:43:24.303ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 335.10637ms iteration = 20 status = {}
92102025-03-07T22:43:24.329ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92112025-03-07T22:43:24.331ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
92122025-03-07T22:43:24.331ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
92132025-03-07T22:43:24.331ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
92142025-03-07T22:43:24.347ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
92152025-03-07T22:43:24.348ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "489cdaf6-b48f-49e8-8ee6-69d995b45cac", "content-length": "1", "date": "Fri, 07 Mar 2025 22:43:23 GMT"} })
92162025-03-07T22:43:24.350ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
92172025-03-07T22:43:24.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
92182025-03-07T22:43:24.351ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 383.102421ms iteration = 20 status = null
92192025-03-07T22:43:24.419ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92202025-03-07T22:43:24.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92212025-03-07T22:43:24.422ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
92222025-03-07T22:43:24.422ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92232025-03-07T22:43:24.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92242025-03-07T22:43:24.451ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
92252025-03-07T22:43:24.452ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd8bf002-e3c0-4c94-b49c-189be752b285", "content-length": "691", "date": "Fri, 07 Mar 2025 22:43:24 GMT"} })
92262025-03-07T22:43:24.452ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
92272025-03-07T22:43:24.452ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
92282025-03-07T22:43:24.452ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
92292025-03-07T22:43:24.454ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 485.511136ms iteration = 20 status = {}
92302025-03-07T22:43:24.467ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
92312025-03-07T22:43:24.467ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "72fda6e5-6b0a-4a24-881c-9a88b9e90ae0", "content-length": "134", "date": "Fri, 07 Mar 2025 22:43:23 GMT"} })
92322025-03-07T22:43:24.467ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
92332025-03-07T22:43:24.467ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
92342025-03-07T22:43:24.467ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {}}
92352025-03-07T22:43:24.467ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
92362025-03-07T22:43:24.467ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
92372025-03-07T22:43:24.467ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {}
92382025-03-07T22:43:24.467ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
92392025-03-07T22:43:24.467ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
92402025-03-07T22:43:24.468ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
92412025-03-07T22:43:24.468ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92422025-03-07T22:43:24.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92432025-03-07T22:43:24.471ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
92442025-03-07T22:43:24.471ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6b9ec06-4595-4c23-8143-d2d312e00848", "content-length": "2", "date": "Fri, 07 Mar 2025 22:43:23 GMT"} })
92452025-03-07T22:43:24.471ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
92462025-03-07T22:43:24.471ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "a975a8d5-f633-4813-9b88-c9af57b4364f", "content-length": "52", "date": "Fri, 07 Mar 2025 22:43:23 GMT"} })
92472025-03-07T22:43:24.471ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
92482025-03-07T22:43:24.471ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
92492025-03-07T22:43:24.679ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 301 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 8853114d-636c-4eca-b22f-874b38c9853c response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
92502025-03-07T22:43:24.699ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "6dd4b55e-baa4-4d60-9645-23945c997d5c", "date": "Fri, 07 Mar 2025 22:43:24 GMT"} })
92512025-03-07T22:43:24.699ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
92522025-03-07T22:43:24.699ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
92532025-03-07T22:43:24.699ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c9eb36f-9a18-4e6a-b514-9eb57e0db906", "content-length": "682", "date": "Fri, 07 Mar 2025 22:43:24 GMT"} })
92542025-03-07T22:43:24.699ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92552025-03-07T22:43:24.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92562025-03-07T22:43:24.703ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92572025-03-07T22:43:24.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92582025-03-07T22:43:24.706ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92592025-03-07T22:43:24.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92602025-03-07T22:43:24.709ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92612025-03-07T22:43:24.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92622025-03-07T22:43:24.716ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 747.669974ms iteration = 20 status = {}
92632025-03-07T22:43:33.969ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 59 reason = Timeout
92642025-03-07T22:43:33.969ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 59 reason = Timeout
92652025-03-07T22:43:33.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92662025-03-07T22:43:33.969ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
92672025-03-07T22:43:33.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92682025-03-07T22:43:33.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92692025-03-07T22:43:33.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92702025-03-07T22:43:33.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
92712025-03-07T22:43:33.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92722025-03-07T22:43:33.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92732025-03-07T22:43:33.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92742025-03-07T22:43:33.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92752025-03-07T22:43:33.999ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
92762025-03-07T22:43:34.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92772025-03-07T22:43:34.006ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92782025-03-07T22:43:34.009ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
92792025-03-07T22:43:34.009ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92802025-03-07T22:43:34.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92812025-03-07T22:43:34.022ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
92822025-03-07T22:43:34.022ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 52.703796ms iteration = 59 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
92832025-03-07T22:43:34.028ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.218019ms iteration = 59 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
92842025-03-07T22:43:34.679ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 331 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 74f58e27-e26d-4d3f-8b51-1c2c303b9c6a response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
92852025-03-07T22:43:43.969ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 60 reason = Timeout
92862025-03-07T22:43:43.969ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 60 reason = Timeout
92872025-03-07T22:43:43.969ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
92882025-03-07T22:43:43.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92892025-03-07T22:43:43.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92902025-03-07T22:43:43.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92912025-03-07T22:43:43.972ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
92922025-03-07T22:43:43.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92932025-03-07T22:43:44.000ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92942025-03-07T22:43:44.002ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.396392ms iteration = 60 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
92952025-03-07T22:43:44.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
92962025-03-07T22:43:44.003ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92972025-03-07T22:43:44.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92982025-03-07T22:43:44.005ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92992025-03-07T22:43:44.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93002025-03-07T22:43:44.007ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93012025-03-07T22:43:44.009ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
93022025-03-07T22:43:44.009ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93032025-03-07T22:43:44.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93042025-03-07T22:43:44.021ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
93052025-03-07T22:43:44.021ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 51.541855ms iteration = 60 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
93062025-03-07T22:43:44.679ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 273 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 3eb23c4a-f3ab-405c-add8-216af763383b response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
93072025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 61 reason = Timeout
93082025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 4 reason = Timeout
93092025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93102025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
93112025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
93122025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
93132025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
93142025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
93152025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
93162025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 21 reason = Timeout
93172025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
93182025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 21 reason = Timeout
93192025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 21 reason = Timeout
93202025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
93212025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
93222025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 21 reason = Timeout
93232025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
93242025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
93252025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
93262025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 21 reason = Timeout
93272025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
93282025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 12 reason = Timeout
93292025-03-07T22:43:53.969ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
93302025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
93312025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.66\u{b5}s iteration = 12 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
93322025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
93332025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 21 reason = Timeout
93342025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Timeout
93352025-03-07T22:43:53.969ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
93362025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93372025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 21 reason = Timeout
93382025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 11 reason = Timeout
93392025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
93402025-03-07T22:43:53.970ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:33:53.969937068 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
93412025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 3 reason = Timeout
93422025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
93432025-03-07T22:43:53.970ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
93442025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 11 reason = Timeout
93452025-03-07T22:43:53.970ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
93462025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93472025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 11 reason = Timeout
93482025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
93492025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
93502025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 21 reason = Timeout
93512025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
93522025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 21 reason = Timeout
93532025-03-07T22:43:53.970ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
93542025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 21 reason = Timeout
93552025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
93562025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
93572025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 61 reason = Timeout
93582025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
93592025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 21 reason = Timeout
93602025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
93612025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 22 reason = Timeout
93622025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
93632025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
93642025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
93652025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
93662025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 21 reason = Timeout
93672025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
93682025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 21 reason = Timeout
93692025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
93702025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
93712025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 2 reason = Timeout
93722025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 11 reason = Timeout
93732025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
93742025-03-07T22:43:53.970ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
93752025-03-07T22:43:53.970ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
93762025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 11 reason = Timeout
93772025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 21 reason = Timeout
93782025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 3 reason = Timeout
93792025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
93802025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
93812025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
93822025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
93832025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 21 reason = Timeout
93842025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 21 reason = Timeout
93852025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
93862025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93872025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
93882025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
93892025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
93902025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
93912025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
93922025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
93932025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93942025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
93952025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
93962025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
93972025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
93982025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
93992025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94002025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
94012025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
94022025-03-07T22:43:53.970ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1392, tv_nsec: 319059225 } })
94032025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
94042025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
94052025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
94062025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
94072025-03-07T22:43:53.970ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1392, tv_nsec: 319099635 } })
94082025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
94092025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
94102025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
94112025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
94122025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
94132025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
94142025-03-07T22:43:53.970ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
94152025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
94162025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
94172025-03-07T22:43:53.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
94182025-03-07T22:43:53.971ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
94192025-03-07T22:43:53.971ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.386253ms iteration = 11 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
94202025-03-07T22:43:53.971ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
94212025-03-07T22:43:53.971ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.590965ms iteration = 11 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
94222025-03-07T22:43:53.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94232025-03-07T22:43:53.972ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
94242025-03-07T22:43:53.972ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1392, tv_nsec: 321206086 } })
94252025-03-07T22:43:53.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
94262025-03-07T22:43:53.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94272025-03-07T22:43:53.973ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
94282025-03-07T22:43:53.973ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.424683ms iteration = 21 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
94292025-03-07T22:43:53.975ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1392, tv_nsec: 323391117 } })
94302025-03-07T22:43:53.975ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
94312025-03-07T22:43:53.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
94322025-03-07T22:43:53.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
94332025-03-07T22:43:53.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
94342025-03-07T22:43:53.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
94352025-03-07T22:43:53.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
94362025-03-07T22:43:53.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
94372025-03-07T22:43:53.977ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94382025-03-07T22:43:53.977ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94392025-03-07T22:43:53.979ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:33:53.969937068 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
94402025-03-07T22:43:53.979ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 9.271059ms iteration = 11 status = {"expiration":"2025-03-07T22:33:53.969937068Z","pruned":[]}
94412025-03-07T22:43:53.979ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94422025-03-07T22:43:53.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
94432025-03-07T22:43:53.979ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
94442025-03-07T22:43:53.979ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
94452025-03-07T22:43:53.979ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
94462025-03-07T22:43:53.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
94472025-03-07T22:43:53.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
94482025-03-07T22:43:53.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
94492025-03-07T22:43:53.981ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
94502025-03-07T22:43:53.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
94512025-03-07T22:43:53.988ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
94522025-03-07T22:43:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
94532025-03-07T22:43:53.989ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94542025-03-07T22:43:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94552025-03-07T22:43:53.994ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 24.560285ms iteration = 11 status = {"errors":[],"found_port_ok":[]}
94562025-03-07T22:43:54.018ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
94572025-03-07T22:43:54.018ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
94582025-03-07T22:43:54.018ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
94592025-03-07T22:43:54.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
94602025-03-07T22:43:54.025ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94612025-03-07T22:43:54.027ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94622025-03-07T22:43:54.027ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94632025-03-07T22:43:54.027ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
94642025-03-07T22:43:54.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
94652025-03-07T22:43:54.029ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
94662025-03-07T22:43:54.029ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94672025-03-07T22:43:54.029ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
94682025-03-07T22:43:54.029ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
94692025-03-07T22:43:54.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94702025-03-07T22:43:54.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
94712025-03-07T22:43:54.031ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
94722025-03-07T22:43:54.031ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
94732025-03-07T22:43:54.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
94742025-03-07T22:43:54.031ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 61.266918ms iteration = 21 status = {"errors":[],"garbage_collect_requested":[]}
94752025-03-07T22:43:54.032ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
94762025-03-07T22:43:54.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
94772025-03-07T22:43:54.034ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
94782025-03-07T22:43:54.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
94792025-03-07T22:43:54.036ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
94802025-03-07T22:43:54.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
94812025-03-07T22:43:54.040ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94822025-03-07T22:43:54.040ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94832025-03-07T22:43:54.042ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94842025-03-07T22:43:54.042ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
94852025-03-07T22:43:54.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
94862025-03-07T22:43:54.042ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 72.549716ms iteration = 21 status = {"errors":[],"finish_invoked_ok":[]}
94872025-03-07T22:43:54.042ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
94882025-03-07T22:43:54.042ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
94892025-03-07T22:43:54.042ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 72.684488ms iteration = 11 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
94902025-03-07T22:43:54.042ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
94912025-03-07T22:43:54.042ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
94922025-03-07T22:43:54.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
94932025-03-07T22:43:54.043ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
94942025-03-07T22:43:54.043ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
94952025-03-07T22:43:54.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
94962025-03-07T22:43:54.044ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
94972025-03-07T22:43:54.044ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
94982025-03-07T22:43:54.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
94992025-03-07T22:43:54.047ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95002025-03-07T22:43:54.047ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95012025-03-07T22:43:54.048ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95022025-03-07T22:43:54.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95032025-03-07T22:43:54.048ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
95042025-03-07T22:43:54.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
95052025-03-07T22:43:54.050ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
95062025-03-07T22:43:54.050ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
95072025-03-07T22:43:54.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
95082025-03-07T22:43:54.052ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
95092025-03-07T22:43:54.054ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
95102025-03-07T22:43:54.054ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 84.609553ms iteration = 2 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
95112025-03-07T22:43:54.055ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
95122025-03-07T22:43:54.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
95132025-03-07T22:43:54.055ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95142025-03-07T22:43:54.059ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
95152025-03-07T22:43:54.060ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
95162025-03-07T22:43:54.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
95172025-03-07T22:43:54.068ZDEBGnexus (BackgroundTasks): inventory_prune_one: eligible for removal background_task = inventory_collection candidates = [(45c4a5be-fd1a-455b-81bd-5dcb75c25a52, 0), (5d300951-e95b-4f3b-b22b-88ab7d93f6e5, 0), (cf1a8eaa-7b99-4282-833e-62dee4622cf9, 0), (24b34bb5-d123-4c98-bdb5-62167bdbc6c1, 0)] collection_id = 45c4a5be-fd1a-455b-81bd-5dcb75c25a52
95182025-03-07T22:43:54.068ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95192025-03-07T22:43:54.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95202025-03-07T22:43:54.072ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 102.360893ms iteration = 21 status = {"errors":[],"requests_created_ok":[]}
95212025-03-07T22:43:54.073ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1392, tv_nsec: 421351678 } })
95222025-03-07T22:43:54.073ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
95232025-03-07T22:43:54.074ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95242025-03-07T22:43:54.075ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
95252025-03-07T22:43:54.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
95262025-03-07T22:43:54.076ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
95272025-03-07T22:43:54.077ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95282025-03-07T22:43:54.077ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95292025-03-07T22:43:54.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95302025-03-07T22:43:54.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95312025-03-07T22:43:54.084ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
95322025-03-07T22:43:54.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
95332025-03-07T22:43:54.087ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
95342025-03-07T22:43:54.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
95352025-03-07T22:43:54.090ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
95362025-03-07T22:43:54.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
95372025-03-07T22:43:54.093ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
95382025-03-07T22:43:54.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
95392025-03-07T22:43:54.094ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
95402025-03-07T22:43:54.094ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 125.00256ms iteration = 21 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
95412025-03-07T22:43:54.095ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95422025-03-07T22:43:54.097ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95432025-03-07T22:43:54.097ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95442025-03-07T22:43:54.100ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
95452025-03-07T22:43:54.103ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95462025-03-07T22:43:54.106ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
95472025-03-07T22:43:54.106ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
95482025-03-07T22:43:54.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
95492025-03-07T22:43:54.107ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
95502025-03-07T22:43:54.107ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
95512025-03-07T22:43:54.107ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
95522025-03-07T22:43:54.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
95532025-03-07T22:43:54.107ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
95542025-03-07T22:43:54.108ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
95552025-03-07T22:43:54.110ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95562025-03-07T22:43:54.111ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95572025-03-07T22:43:54.111ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95582025-03-07T22:43:54.112ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 142.958423ms iteration = 61 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
95592025-03-07T22:43:54.113ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
95602025-03-07T22:43:54.113ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
95612025-03-07T22:43:54.114ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
95622025-03-07T22:43:54.114ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
95632025-03-07T22:43:54.114ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 144.575019ms iteration = 61 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
95642025-03-07T22:43:54.114ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
95652025-03-07T22:43:54.114ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
95662025-03-07T22:43:54.114ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
95672025-03-07T22:43:54.114ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
95682025-03-07T22:43:54.114ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
95692025-03-07T22:43:54.116ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
95702025-03-07T22:43:54.116ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95712025-03-07T22:43:54.118ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95722025-03-07T22:43:54.118ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
95732025-03-07T22:43:54.118ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
95742025-03-07T22:43:54.118ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 148.767009ms iteration = 11 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
95752025-03-07T22:43:54.119ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
95762025-03-07T22:43:54.119ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
95772025-03-07T22:43:54.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
95782025-03-07T22:43:54.122ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 152.518245ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
95792025-03-07T22:43:54.122ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95802025-03-07T22:43:54.123ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
95812025-03-07T22:43:54.123ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
95822025-03-07T22:43:54.123ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95832025-03-07T22:43:54.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
95842025-03-07T22:43:54.125ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
95852025-03-07T22:43:54.125ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
95862025-03-07T22:43:54.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
95872025-03-07T22:43:54.128ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
95882025-03-07T22:43:54.128ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
95892025-03-07T22:43:54.128ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
95902025-03-07T22:43:54.131ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
95912025-03-07T22:43:54.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
95922025-03-07T22:43:54.132ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
95932025-03-07T22:43:54.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
95942025-03-07T22:43:54.135ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
95952025-03-07T22:43:54.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
95962025-03-07T22:43:54.135ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95972025-03-07T22:43:54.137ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95982025-03-07T22:43:54.139ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
95992025-03-07T22:43:54.139ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
96002025-03-07T22:43:54.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
96012025-03-07T22:43:54.141ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
96022025-03-07T22:43:54.141ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
96032025-03-07T22:43:54.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
96042025-03-07T22:43:54.157ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96052025-03-07T22:43:54.158ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 188.388149ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
96062025-03-07T22:43:54.160ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 190.800294ms iteration = 21 status = 0
96072025-03-07T22:43:54.161ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
96082025-03-07T22:43:54.161ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
96092025-03-07T22:43:54.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
96102025-03-07T22:43:54.171ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96112025-03-07T22:43:54.172ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
96122025-03-07T22:43:54.172ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 203.140382ms iteration = 4 status = {"blueprint_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","inventory_collection_id":"24b34bb5-d123-4c98-bdb5-62167bdbc6c1","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
96132025-03-07T22:43:54.176ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
96142025-03-07T22:43:54.176ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
96152025-03-07T22:43:54.177ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
96162025-03-07T22:43:54.177ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96172025-03-07T22:43:54.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96182025-03-07T22:43:54.177ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 207.403173ms iteration = 21 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
96192025-03-07T22:43:54.178ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
96202025-03-07T22:43:54.178ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
96212025-03-07T22:43:54.179ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "59fcbe2c-5e15-4f0c-a7d7-57d99194eb2a", "content-length": "26", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
96222025-03-07T22:43:54.179ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
96232025-03-07T22:43:54.179ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96242025-03-07T22:43:54.180ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96252025-03-07T22:43:54.181ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
96262025-03-07T22:43:54.182ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96272025-03-07T22:43:54.182ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96282025-03-07T22:43:54.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96292025-03-07T22:43:54.184ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
96302025-03-07T22:43:54.184ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
96312025-03-07T22:43:54.184ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 214.670863ms iteration = 21 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
96322025-03-07T22:43:54.185ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
96332025-03-07T22:43:54.185ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
96342025-03-07T22:43:54.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
96352025-03-07T22:43:54.187ZINFOnexus (BackgroundTasks): removed inventory collection background_task = inventory_collection collection_id = 45c4a5be-fd1a-455b-81bd-5dcb75c25a52 file = nexus/db-queries/src/db/datastore/inventory.rs:1442 ncabooses = 12 nclickhouse_keeper_membership = 0 ncollections = 1 ndatasets = 0 nerrors = 0 nnics = 0 nnvme_disk_firmware = 0 nphysical_disks = 0 nrot_pages = 8 nrots = 2 nsled_agent_zones = 0 nsled_agents = 0 nsps = 2 nzones = 0 nzpools = 0
96362025-03-07T22:43:54.187ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96372025-03-07T22:43:54.187ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96382025-03-07T22:43:54.187ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96392025-03-07T22:43:54.191ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5d300951-e95b-4f3b-b22b-88ab7d93f6e5, 0), (cf1a8eaa-7b99-4282-833e-62dee4622cf9, 0), (24b34bb5-d123-4c98-bdb5-62167bdbc6c1, 0)]
96402025-03-07T22:43:54.192ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1392, tv_nsec: 540561903 } })
96412025-03-07T22:43:54.192ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
96422025-03-07T22:43:54.193ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96432025-03-07T22:43:54.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96442025-03-07T22:43:54.196ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96452025-03-07T22:43:54.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96462025-03-07T22:43:54.198ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
96472025-03-07T22:43:54.198ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
96482025-03-07T22:43:54.198ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
96492025-03-07T22:43:54.198ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
96502025-03-07T22:43:54.198ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96512025-03-07T22:43:54.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
96522025-03-07T22:43:54.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96532025-03-07T22:43:54.201ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96542025-03-07T22:43:54.202ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96552025-03-07T22:43:54.202ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96562025-03-07T22:43:54.202ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
96572025-03-07T22:43:54.202ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
96582025-03-07T22:43:54.202ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
96592025-03-07T22:43:54.203ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
96602025-03-07T22:43:54.204ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
96612025-03-07T22:43:54.204ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
96622025-03-07T22:43:54.204ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
96632025-03-07T22:43:54.204ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96642025-03-07T22:43:54.205ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96652025-03-07T22:43:54.205ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96662025-03-07T22:43:54.208ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
96672025-03-07T22:43:54.208ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
96682025-03-07T22:43:54.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
96692025-03-07T22:43:54.209ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
96702025-03-07T22:43:54.209ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
96712025-03-07T22:43:54.210ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96722025-03-07T22:43:54.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96732025-03-07T22:43:54.211ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96742025-03-07T22:43:54.211ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
96752025-03-07T22:43:54.211ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
96762025-03-07T22:43:54.211ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-07 22:34:17.797135 UTC dns_group = external generation = 2
96772025-03-07T22:43:54.211ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 241.46366ms iteration = 11 status = {"generation":2}
96782025-03-07T22:43:54.212ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96792025-03-07T22:43:54.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96802025-03-07T22:43:54.214ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
96812025-03-07T22:43:54.214ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
96822025-03-07T22:43:54.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
96832025-03-07T22:43:54.220ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96842025-03-07T22:43:54.227ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
96852025-03-07T22:43:54.227ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96862025-03-07T22:43:54.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96872025-03-07T22:43:54.230ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96882025-03-07T22:43:54.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96892025-03-07T22:43:54.232ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96902025-03-07T22:43:54.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96912025-03-07T22:43:54.235ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96922025-03-07T22:43:54.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96932025-03-07T22:43:54.239ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96942025-03-07T22:43:54.239ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96952025-03-07T22:43:54.245ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
96962025-03-07T22:43:54.245ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 275.87488ms iteration = 21 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
96972025-03-07T22:43:54.245ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96982025-03-07T22:43:54.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96992025-03-07T22:43:54.248ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
97002025-03-07T22:43:54.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
97012025-03-07T22:43:54.250ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97022025-03-07T22:43:54.256ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
97032025-03-07T22:43:54.256ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
97042025-03-07T22:43:54.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
97052025-03-07T22:43:54.262ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
97062025-03-07T22:43:54.263ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
97072025-03-07T22:43:54.263ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
97082025-03-07T22:43:54.263ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f926d359-70dd-4617-9284-1d7a1787fbfd", "content-length": "26", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97092025-03-07T22:43:54.263ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
97102025-03-07T22:43:54.265ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
97112025-03-07T22:43:54.265ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
97122025-03-07T22:43:54.265ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
97132025-03-07T22:43:54.265ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
97142025-03-07T22:43:54.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
97152025-03-07T22:43:54.281ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
97162025-03-07T22:43:54.281ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 311.1602ms iteration = 11 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
97172025-03-07T22:43:54.287ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) }
97182025-03-07T22:43:54.287ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 317.730812ms iteration = 11 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
97192025-03-07T22:43:54.288ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
97202025-03-07T22:43:54.288ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
97212025-03-07T22:43:54.288ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3feba2b4-b71a-47ff-b924-eb4403a8aa53", "content-length": "26", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97222025-03-07T22:43:54.288ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
97232025-03-07T22:43:54.294ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
97242025-03-07T22:43:54.303ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 333.550485ms iteration = 22 status = {"physical_disks_added":0}
97252025-03-07T22:43:54.327ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
97262025-03-07T22:43:54.327ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
97272025-03-07T22:43:54.357ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
97282025-03-07T22:43:54.358ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "41f2e889-8b96-495f-a9cb-e51859ede7de", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
97292025-03-07T22:43:54.358ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
97302025-03-07T22:43:54.360ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "2bf4ca69-e687-436b-a571-2ba807f77553", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
97312025-03-07T22:43:54.360ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
97322025-03-07T22:43:54.362ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
97332025-03-07T22:43:54.362ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 392.36818ms iteration = 13 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
97342025-03-07T22:43:54.384ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 414.288601ms iteration = 21 status = {}
97352025-03-07T22:43:54.401ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
97362025-03-07T22:43:54.401ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
97372025-03-07T22:43:54.401ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
97382025-03-07T22:43:54.401ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7018f57-906a-4682-b70d-a2c78d7cd641", "content-length": "476", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97392025-03-07T22:43:54.401ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
97402025-03-07T22:43:54.401ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f0ca217-bd7a-4ea2-a51e-ea8a4ea46ac1", "content-length": "734", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97412025-03-07T22:43:54.401ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
97422025-03-07T22:43:54.402ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "66aabb94-79e7-4aff-bd95-a8de394708fb", "content-length": "111", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97432025-03-07T22:43:54.402ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
97442025-03-07T22:43:54.402ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a65de14-e790-454c-96f3-c7c6fd8753a9", "content-length": "111", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97452025-03-07T22:43:54.402ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
97462025-03-07T22:43:54.402ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9b3ba29-29c3-4f89-8ee1-ba7f3ecc5225", "content-length": "167", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97472025-03-07T22:43:54.402ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
97482025-03-07T22:43:54.403ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2441b47a-f18d-4d2c-8e44-d90ac6b1bf13", "content-length": "167", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97492025-03-07T22:43:54.403ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
97502025-03-07T22:43:54.403ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52c4b84d-0e1b-4188-b4d2-11755cf60516", "content-length": "175", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97512025-03-07T22:43:54.403ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
97522025-03-07T22:43:54.403ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e5fb650-a397-4bdb-a765-f719523078a4", "content-length": "175", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97532025-03-07T22:43:54.403ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
97542025-03-07T22:43:54.403ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d382738b-6bbc-4142-923d-b94815a05b43", "content-length": "702", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97552025-03-07T22:43:54.403ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
97562025-03-07T22:43:54.403ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a550302-c00e-41b2-bca5-a62cd608c66f", "content-length": "727", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97572025-03-07T22:43:54.403ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
97582025-03-07T22:43:54.404ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19dac152-fc27-48ae-9064-f721be9f3fbb", "content-length": "729", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97592025-03-07T22:43:54.404ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
97602025-03-07T22:43:54.404ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1737f23a-44d8-4a59-97c1-6282c6ae7150", "content-length": "728", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97612025-03-07T22:43:54.404ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
97622025-03-07T22:43:54.404ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b83bf64-d119-4601-ad7d-2cb73740667e", "content-length": "722", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97632025-03-07T22:43:54.404ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
97642025-03-07T22:43:54.404ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b308c73-3152-4bf7-a426-86366dca6aaf", "content-length": "109", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97652025-03-07T22:43:54.404ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
97662025-03-07T22:43:54.404ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48b97071-5f68-402c-bdf2-8ee7e7006c4c", "content-length": "109", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97672025-03-07T22:43:54.404ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
97682025-03-07T22:43:54.405ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cce915eb-6182-4323-85c9-a47ee1dd844d", "content-length": "169", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97692025-03-07T22:43:54.405ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
97702025-03-07T22:43:54.405ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db104a54-0f4f-41b7-9462-fbb973693d9a", "content-length": "169", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97712025-03-07T22:43:54.405ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
97722025-03-07T22:43:54.405ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04bbefe1-424d-42d2-9ade-35f38d3c5f33", "content-length": "178", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97732025-03-07T22:43:54.405ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
97742025-03-07T22:43:54.405ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b591a14-58e4-495d-a213-b5b0ec17d88b", "content-length": "178", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97752025-03-07T22:43:54.405ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
97762025-03-07T22:43:54.406ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfbc45f5-6f27-4567-9ca8-56c8ca0ebfd4", "content-length": "702", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97772025-03-07T22:43:54.406ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
97782025-03-07T22:43:54.406ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "339121e5-4bf9-4620-8a95-5cc9e676faee", "content-length": "727", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97792025-03-07T22:43:54.406ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
97802025-03-07T22:43:54.406ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "858b0ee8-9c01-4314-bd31-dc5c287250b2", "content-length": "729", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97812025-03-07T22:43:54.406ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
97822025-03-07T22:43:54.406ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96d64fae-8698-46b3-9ec4-732239f49ef0", "content-length": "728", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97832025-03-07T22:43:54.406ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97842025-03-07T22:43:54.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97852025-03-07T22:43:54.420ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
97862025-03-07T22:43:54.420ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
97872025-03-07T22:43:54.420ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "2602ac36-a432-4584-9ad1-df82d402222f", "content-length": "372", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
97882025-03-07T22:43:54.421ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
97892025-03-07T22:43:54.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
97902025-03-07T22:43:54.433ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
97912025-03-07T22:43:54.433ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
97922025-03-07T22:43:54.435ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
97932025-03-07T22:43:54.435ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
97942025-03-07T22:43:54.436ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97952025-03-07T22:43:54.436ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97962025-03-07T22:43:54.436ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97972025-03-07T22:43:54.438ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97982025-03-07T22:43:54.438ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
97992025-03-07T22:43:54.439ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
98002025-03-07T22:43:54.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
98012025-03-07T22:43:54.439ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
98022025-03-07T22:43:54.439ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98032025-03-07T22:43:54.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98042025-03-07T22:43:54.440ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
98052025-03-07T22:43:54.440ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
98062025-03-07T22:43:54.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
98072025-03-07T22:43:54.441ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
98082025-03-07T22:43:54.441ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
98092025-03-07T22:43:54.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
98102025-03-07T22:43:54.442ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
98112025-03-07T22:43:54.442ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
98122025-03-07T22:43:54.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
98132025-03-07T22:43:54.442ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98142025-03-07T22:43:54.445ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
98152025-03-07T22:43:54.445ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
98162025-03-07T22:43:54.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
98172025-03-07T22:43:54.447ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98182025-03-07T22:43:54.450ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
98192025-03-07T22:43:54.450ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
98202025-03-07T22:43:54.450ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-07 22:34:17.208516 UTC dns_group = internal generation = 1
98212025-03-07T22:43:54.450ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 480.980832ms iteration = 11 status = {"generation":1}
98222025-03-07T22:43:54.452ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
98232025-03-07T22:43:54.452ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98242025-03-07T22:43:54.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98252025-03-07T22:43:54.452ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98262025-03-07T22:43:54.453ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
98272025-03-07T22:43:54.454ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ce4b80f-0e78-4f35-81b5-5e91999ad91c", "content-length": "2", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
98282025-03-07T22:43:54.454ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:615 sled = http://[fd00:1122:3344:101::1]:12345
98292025-03-07T22:43:54.456ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
98302025-03-07T22:43:54.457ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ef1ef21-96c6-4f0c-8160-8d5de82a6fca", "content-length": "1", "date": "Fri, 07 Mar 2025 22:43:53 GMT"} })
98312025-03-07T22:43:54.457ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 487.373253ms iteration = 3 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":2,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"73b218dd-7dbc-4345-acb1-fe69ced4a7f7","time":"2025-03-07T22:38:54.488827469Z"},{"operation":"list","target_sled":"73b218dd-7dbc-4345-acb1-fe69ced4a7f7","time":"2025-03-07T22:43:54.454396963Z"}]}
98322025-03-07T22:43:54.459ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
98332025-03-07T22:43:54.459ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
98342025-03-07T22:43:54.459ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
98352025-03-07T22:43:54.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
98362025-03-07T22:43:54.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
98372025-03-07T22:43:54.462ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 492.741075ms iteration = 21 status = null
98382025-03-07T22:43:54.474ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
98392025-03-07T22:43:54.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
98402025-03-07T22:43:54.488ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
98412025-03-07T22:43:54.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
98422025-03-07T22:43:54.491ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
98432025-03-07T22:43:54.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
98442025-03-07T22:43:54.513ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
98452025-03-07T22:43:54.513ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
98462025-03-07T22:43:54.514ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "f15f27a5-927c-483d-ace7-459a9f7e2b4b", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
98472025-03-07T22:43:54.514ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98482025-03-07T22:43:54.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98492025-03-07T22:43:54.575ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
98502025-03-07T22:43:54.576ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "4a2fc72b-587b-4a81-a3e5-b455f90251f5", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
98512025-03-07T22:43:54.576ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
98522025-03-07T22:43:54.577ZDEBGnexus (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[fd00:1122:3344:101::1]:12345
98532025-03-07T22:43:54.577ZDEBGnexus (BackgroundTasks): client request SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/inventory
98542025-03-07T22:43:54.577ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "4c25bcc3-78de-4cef-9d86-6e2e1cc93b28", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
98552025-03-07T22:43:54.577ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
98562025-03-07T22:43:54.579ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "8ef77a26-95e2-4450-a6a4-5875b5d678cd", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
98572025-03-07T22:43:54.579ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
98582025-03-07T22:43:54.581ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
98592025-03-07T22:43:54.581ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 611.846109ms iteration = 13 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
98602025-03-07T22:43:54.582ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
98612025-03-07T22:43:54.583ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
98622025-03-07T22:43:54.583ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98632025-03-07T22:43:54.583ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98642025-03-07T22:43:54.586ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
98652025-03-07T22:43:54.586ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
98662025-03-07T22:43:54.587ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98672025-03-07T22:43:54.587ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98682025-03-07T22:43:54.621ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
98692025-03-07T22:43:54.621ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fcae382-2ee8-4d96-98e6-1cc7a29dfa8a", "content-length": "691", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
98702025-03-07T22:43:54.622ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
98712025-03-07T22:43:54.622ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
98722025-03-07T22:43:54.622ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
98732025-03-07T22:43:54.623ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 653.266238ms iteration = 21 status = {}
98742025-03-07T22:43:54.625ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
98752025-03-07T22:43:54.625ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
98762025-03-07T22:43:54.628ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "3da79c89-defc-4032-ac0b-49d1a5592630", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
98772025-03-07T22:43:54.630ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 660.595318ms iteration = 21 status = {}
98782025-03-07T22:43:54.636ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
98792025-03-07T22:43:54.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
98802025-03-07T22:43:54.638ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
98812025-03-07T22:43:54.638ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
98822025-03-07T22:43:54.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
98832025-03-07T22:43:54.648ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
98842025-03-07T22:43:54.648ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f164c15-bfca-4dc3-a858-2c875822e2fa", "content-length": "134", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
98852025-03-07T22:43:54.648ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
98862025-03-07T22:43:54.648ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
98872025-03-07T22:43:54.649ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
98882025-03-07T22:43:54.649ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
98892025-03-07T22:43:54.649ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
98902025-03-07T22:43:54.649ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
98912025-03-07T22:43:54.649ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
98922025-03-07T22:43:54.649ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
98932025-03-07T22:43:54.649ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
98942025-03-07T22:43:54.649ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
98952025-03-07T22:43:54.649ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
98962025-03-07T22:43:54.649ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
98972025-03-07T22:43:54.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
98982025-03-07T22:43:54.651ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
98992025-03-07T22:43:54.651ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "4baa901e-31f4-443e-aa13-53b3c3e3ff09", "content-length": "2", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
99002025-03-07T22:43:54.652ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
99012025-03-07T22:43:54.652ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d127d4d-628e-4240-8d47-0f30f0403c42", "content-length": "52", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
99022025-03-07T22:43:54.652ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
99032025-03-07T22:43:54.652ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
99042025-03-07T22:43:54.665ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
99052025-03-07T22:43:54.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
99062025-03-07T22:43:54.667ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
99072025-03-07T22:43:54.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
99082025-03-07T22:43:54.669ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99092025-03-07T22:43:54.673ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
99102025-03-07T22:43:54.673ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
99112025-03-07T22:43:54.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
99122025-03-07T22:43:54.675ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
99132025-03-07T22:43:54.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
99142025-03-07T22:43:54.677ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
99152025-03-07T22:43:54.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
99162025-03-07T22:43:54.679ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 331 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 4483a25e-3d58-4a7e-82e3-6ef120474bba response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
99172025-03-07T22:43:54.680ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99182025-03-07T22:43:54.685ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
99192025-03-07T22:43:54.685ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
99202025-03-07T22:43:54.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
99212025-03-07T22:43:54.696ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 867d8023-aff2-4521-891f-07d5f45be915, kind: Service { id: dfe169a3-7536-4b54-9593-8815d6e81dd4 }, name: Name("nexus-dfe169a3-7536-4b54-9593-8815d6e81dd4"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 239])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8980b304-809c-4233-b9df-dd6e12e7c2f3, kind: Service { id: c0f69897-7f92-43db-ad47-96c3c4721128 }, name: Name("nexus-c0f69897-7f92-43db-ad47-96c3c4721128"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 221, 186])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: cb82146b-41bd-465f-8803-0f44afcd8f6e, kind: Service { id: 0f5c512c-2fda-4cff-bbad-41e8a283d9ff }, name: Name("nexus-0f5c512c-2fda-4cff-bbad-41e8a283d9ff"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 140, 207])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 00c422b8-3a81-4d2c-bc3f-fc426f6f9897, kind: Service { id: fbf58a59-609b-4fb3-b792-2c09f3f7b776 }, name: Name("external-dns-fbf58a59-609b-4fb3-b792-2c09f3f7b776"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 149, 171])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 066589e5-6cdf-4b01-9f39-98f2c968e6c2, kind: Service { id: 732b0e55-5ccf-4068-b771-1299f872f618 }, name: Name("external-dns-732b0e55-5ccf-4068-b771-1299f872f618"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
99222025-03-07T22:43:54.696ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
99232025-03-07T22:43:54.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
99242025-03-07T22:43:54.698ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99252025-03-07T22:43:54.700ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
99262025-03-07T22:43:54.700ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
99272025-03-07T22:43:54.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
99282025-03-07T22:43:54.702ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
99292025-03-07T22:43:54.702ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 00c422b8-3a81-4d2c-bc3f-fc426f6f9897, kind: Service { id: fbf58a59-609b-4fb3-b792-2c09f3f7b776 }, name: Name("external-dns-fbf58a59-609b-4fb3-b792-2c09f3f7b776"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 149, 171])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 066589e5-6cdf-4b01-9f39-98f2c968e6c2, kind: Service { id: 732b0e55-5ccf-4068-b771-1299f872f618 }, name: Name("external-dns-732b0e55-5ccf-4068-b771-1299f872f618"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 242])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 867d8023-aff2-4521-891f-07d5f45be915, kind: Service { id: dfe169a3-7536-4b54-9593-8815d6e81dd4 }, name: Name("nexus-dfe169a3-7536-4b54-9593-8815d6e81dd4"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 192, 239])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8980b304-809c-4233-b9df-dd6e12e7c2f3, kind: Service { id: c0f69897-7f92-43db-ad47-96c3c4721128 }, name: Name("nexus-c0f69897-7f92-43db-ad47-96c3c4721128"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 221, 186])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: cb82146b-41bd-465f-8803-0f44afcd8f6e, kind: Service { id: 0f5c512c-2fda-4cff-bbad-41e8a283d9ff }, name: Name("nexus-0f5c512c-2fda-4cff-bbad-41e8a283d9ff"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 140, 207])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
99302025-03-07T22:43:54.702ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
99312025-03-07T22:43:54.767ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "fe74dae2-a5ba-4c1b-80dd-cc85e04129f4", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
99322025-03-07T22:43:54.768ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
99332025-03-07T22:43:54.768ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
99342025-03-07T22:43:54.772ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad422b49-7ba9-463c-99de-391a99cdb80d", "content-length": "682", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
99352025-03-07T22:43:54.772ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
99362025-03-07T22:43:54.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
99372025-03-07T22:43:54.777ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
99382025-03-07T22:43:54.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
99392025-03-07T22:43:54.782ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
99402025-03-07T22:43:54.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
99412025-03-07T22:43:54.784ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
99422025-03-07T22:43:54.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
99432025-03-07T22:43:54.793ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 823.15996ms iteration = 21 status = {}
99442025-03-07T22:43:54.824ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, time_created: 2025-03-07T22:34:17.183069Z, time_modified: 2025-03-07T22:34:17.183069Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, is_scrimlet: true, serial_number: "buskin", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
99452025-03-07T22:43:54.824ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
99462025-03-07T22:43:54.824ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
99472025-03-07T22:43:54.825ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
99482025-03-07T22:43:54.826ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
99492025-03-07T22:43:54.827ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
99502025-03-07T22:43:54.828ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99512025-03-07T22:43:54.831ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
99522025-03-07T22:43:54.872ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
99532025-03-07T22:43:54.878ZDEBGnexus (BackgroundTasks): client response SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc0309f8-7673-4868-b461-c66dc82a64d5", "content-length": "31978", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
99542025-03-07T22:43:54.878ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "0168455b-fa6f-44b1-a034-b95ef7360d9d", "date": "Fri, 07 Mar 2025 22:43:54 GMT"} })
99552025-03-07T22:43:54.881ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
99562025-03-07T22:43:54.881ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 911.146336ms
99572025-03-07T22:43:54.881ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 911.157326ms iteration = 3 status = {}
99582025-03-07T22:43:54.912ZDEBGnexus (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
99592025-03-07T22:43:54.912ZDEBGnexus (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
99602025-03-07T22:43:54.912ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
99612025-03-07T22:43:54.916ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99622025-03-07T22:43:54.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99632025-03-07T22:43:54.918ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99642025-03-07T22:43:54.920ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99652025-03-07T22:43:54.920ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99662025-03-07T22:43:54.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99672025-03-07T22:43:55.116ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 462ebb38-99f4-43ea-8f01-69f80eb7a1ff file = nexus/db-queries/src/db/datastore/inventory.rs:1010
99682025-03-07T22:43:55.116ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 462ebb38-99f4-43ea-8f01-69f80eb7a1ff time_started = 2025-03-07 22:43:54.401225 UTC
99692025-03-07T22:43:55.116ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 23 reason = Dependency
99702025-03-07T22:43:55.116ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
99712025-03-07T22:43:55.116ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.146588449s iteration = 2 status = {"collection_id":"462ebb38-99f4-43ea-8f01-69f80eb7a1ff","time_done":"2025-03-07 22:43:54.914630 UTC","time_started":"2025-03-07 22:43:54.401225 UTC"}
99722025-03-07T22:43:55.116ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 5 reason = Dependency
99732025-03-07T22:43:55.116ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
99742025-03-07T22:43:55.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
99752025-03-07T22:43:55.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
99762025-03-07T22:43:55.118ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99772025-03-07T22:43:55.119ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99782025-03-07T22:43:55.120ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
99792025-03-07T22:43:55.121ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
99802025-03-07T22:43:55.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
99812025-03-07T22:43:55.122ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
99822025-03-07T22:43:55.122ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
99832025-03-07T22:43:55.122ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
99842025-03-07T22:43:55.123ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
99852025-03-07T22:43:55.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
99862025-03-07T22:43:55.124ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99872025-03-07T22:43:55.124ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
99882025-03-07T22:43:55.125ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
99892025-03-07T22:43:55.127ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
99902025-03-07T22:43:55.127ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
99912025-03-07T22:43:55.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
99922025-03-07T22:43:55.156ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
99932025-03-07T22:43:55.156ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
99942025-03-07T22:43:55.157ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99952025-03-07T22:43:55.159ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
99962025-03-07T22:43:55.159ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
99972025-03-07T22:43:55.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
99982025-03-07T22:43:55.161ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99992025-03-07T22:43:55.163ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
100002025-03-07T22:43:55.163ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
100012025-03-07T22:43:55.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
100022025-03-07T22:43:55.165ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
100032025-03-07T22:43:55.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
100042025-03-07T22:43:55.166ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 50.474305ms iteration = 23 status = {"physical_disks_added":0}
100052025-03-07T22:43:55.166ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100062025-03-07T22:43:55.168ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
100072025-03-07T22:43:55.168ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
100082025-03-07T22:43:55.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
100092025-03-07T22:43:55.172ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100102025-03-07T22:43:55.174ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
100112025-03-07T22:43:55.174ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
100122025-03-07T22:43:55.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
100132025-03-07T22:43:55.176ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
100142025-03-07T22:43:55.176ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 60.040737ms iteration = 5 status = {"blueprint_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","inventory_collection_id":"462ebb38-99f4-43ea-8f01-69f80eb7a1ff","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
100152025-03-07T22:44:03.970ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 62 reason = Timeout
100162025-03-07T22:44:03.970ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
100172025-03-07T22:44:03.970ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 62 reason = Timeout
100182025-03-07T22:44:03.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100192025-03-07T22:44:03.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
100202025-03-07T22:44:03.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100212025-03-07T22:44:03.972ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
100222025-03-07T22:44:03.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
100232025-03-07T22:44:03.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100242025-03-07T22:44:03.974ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.946418ms iteration = 62 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
100252025-03-07T22:44:03.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
100262025-03-07T22:44:03.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100272025-03-07T22:44:03.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100282025-03-07T22:44:03.977ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100292025-03-07T22:44:03.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100302025-03-07T22:44:03.978ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100312025-03-07T22:44:03.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
100322025-03-07T22:44:03.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100332025-03-07T22:44:03.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100342025-03-07T22:44:03.990ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
100352025-03-07T22:44:03.990ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.573338ms iteration = 62 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
100362025-03-07T22:44:04.680ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 261 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 79d0088c-8146-4ab1-a82f-41a69559a246 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
100372025-03-07T22:44:13.970ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 63 reason = Timeout
100382025-03-07T22:44:13.970ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
100392025-03-07T22:44:13.970ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 63 reason = Timeout
100402025-03-07T22:44:13.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100412025-03-07T22:44:13.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
100422025-03-07T22:44:13.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100432025-03-07T22:44:13.972ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
100442025-03-07T22:44:13.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
100452025-03-07T22:44:13.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100462025-03-07T22:44:13.974ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.15656ms iteration = 63 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
100472025-03-07T22:44:13.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
100482025-03-07T22:44:13.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100492025-03-07T22:44:13.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100502025-03-07T22:44:13.977ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100512025-03-07T22:44:13.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100522025-03-07T22:44:13.979ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100532025-03-07T22:44:13.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
100542025-03-07T22:44:13.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100552025-03-07T22:44:13.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100562025-03-07T22:44:13.991ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
100572025-03-07T22:44:13.991ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.004822ms iteration = 63 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
100582025-03-07T22:44:14.680ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 281 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 0220dbfb-2d30-4b5c-9931-6c86e0c4959b response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
100592025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 64 reason = Timeout
100602025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 64 reason = Timeout
100612025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100622025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
100632025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 22 reason = Timeout
100642025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
100652025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 24 reason = Timeout
100662025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
100672025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
100682025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 22 reason = Timeout
100692025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 22 reason = Timeout
100702025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
100712025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 22 reason = Timeout
100722025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
100732025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 22 reason = Timeout
100742025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 22 reason = Timeout
100752025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
100762025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 22 reason = Timeout
100772025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
100782025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
100792025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 22 reason = Timeout
100802025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
100812025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 22 reason = Timeout
100822025-03-07T22:44:23.970ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
100832025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
100842025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 22 reason = Timeout
100852025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
100862025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 22 reason = Timeout
100872025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100882025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
100892025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100902025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
100912025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
100922025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
100932025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
100942025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
100952025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
100962025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
100972025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
100982025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
100992025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
101002025-03-07T22:44:23.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
101012025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 22 reason = Timeout
101022025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 22 reason = Timeout
101032025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
101042025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
101052025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 22 reason = Timeout
101062025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
101072025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 22 reason = Timeout
101082025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
101092025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 22 reason = Timeout
101102025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
101112025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
101122025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
101132025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
101142025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
101152025-03-07T22:44:23.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
101162025-03-07T22:44:23.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101172025-03-07T22:44:23.973ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101182025-03-07T22:44:23.974ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101192025-03-07T22:44:23.974ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.339632ms iteration = 22 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
101202025-03-07T22:44:23.974ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
101212025-03-07T22:44:23.974ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
101222025-03-07T22:44:23.974ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
101232025-03-07T22:44:23.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
101242025-03-07T22:44:23.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
101252025-03-07T22:44:23.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
101262025-03-07T22:44:23.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
101272025-03-07T22:44:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
101282025-03-07T22:44:23.976ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
101292025-03-07T22:44:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
101302025-03-07T22:44:23.976ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
101312025-03-07T22:44:23.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
101322025-03-07T22:44:23.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
101332025-03-07T22:44:23.977ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101342025-03-07T22:44:23.977ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101352025-03-07T22:44:23.978ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101362025-03-07T22:44:23.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
101372025-03-07T22:44:23.979ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
101382025-03-07T22:44:23.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
101392025-03-07T22:44:23.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
101402025-03-07T22:44:23.979ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
101412025-03-07T22:44:23.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
101422025-03-07T22:44:23.980ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
101432025-03-07T22:44:23.980ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
101442025-03-07T22:44:23.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
101452025-03-07T22:44:23.981ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1422, tv_nsec: 328874569 } })
101462025-03-07T22:44:23.981ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
101472025-03-07T22:44:23.981ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101482025-03-07T22:44:23.981ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1422, tv_nsec: 329542836 } })
101492025-03-07T22:44:23.982ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
101502025-03-07T22:44:23.982ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 11.41754ms iteration = 22 status = {"errors":[],"garbage_collect_requested":[]}
101512025-03-07T22:44:23.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
101522025-03-07T22:44:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
101532025-03-07T22:44:23.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
101542025-03-07T22:44:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
101552025-03-07T22:44:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
101562025-03-07T22:44:23.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
101572025-03-07T22:44:23.986ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
101582025-03-07T22:44:23.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
101592025-03-07T22:44:23.986ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
101602025-03-07T22:44:23.986ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
101612025-03-07T22:44:23.986ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
101622025-03-07T22:44:23.986ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
101632025-03-07T22:44:23.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
101642025-03-07T22:44:23.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
101652025-03-07T22:44:23.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
101662025-03-07T22:44:23.987ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
101672025-03-07T22:44:23.987ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101682025-03-07T22:44:23.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
101692025-03-07T22:44:23.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101702025-03-07T22:44:23.989ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
101712025-03-07T22:44:23.989ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
101722025-03-07T22:44:23.989ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1422, tv_nsec: 336982847 } })
101732025-03-07T22:44:23.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
101742025-03-07T22:44:23.989ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
101752025-03-07T22:44:23.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
101762025-03-07T22:44:23.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
101772025-03-07T22:44:23.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
101782025-03-07T22:44:23.992ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101792025-03-07T22:44:23.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
101802025-03-07T22:44:23.994ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
101812025-03-07T22:44:23.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
101822025-03-07T22:44:23.995ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101832025-03-07T22:44:23.997ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
101842025-03-07T22:44:23.997ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 27.513724ms iteration = 22 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
101852025-03-07T22:44:23.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
101862025-03-07T22:44:23.997ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
101872025-03-07T22:44:23.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
101882025-03-07T22:44:24.013ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
101892025-03-07T22:44:24.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
101902025-03-07T22:44:24.015ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 45.181505ms iteration = 64 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
101912025-03-07T22:44:24.016ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 46.399255ms iteration = 24 status = {"physical_disks_added":0}
101922025-03-07T22:44:24.022ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
101932025-03-07T22:44:24.022ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 52.648066ms iteration = 64 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
101942025-03-07T22:44:24.023ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 51.99236ms iteration = 22 status = {"errors":[],"finish_invoked_ok":[]}
101952025-03-07T22:44:24.023ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
101962025-03-07T22:44:24.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
101972025-03-07T22:44:24.025ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 54.12968ms iteration = 22 status = {"errors":[],"requests_created_ok":[]}
101982025-03-07T22:44:24.025ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 55.531494ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
101992025-03-07T22:44:24.028ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
102002025-03-07T22:44:24.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
102012025-03-07T22:44:24.028ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
102022025-03-07T22:44:24.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
102032025-03-07T22:44:24.030ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102042025-03-07T22:44:24.030ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 58.875746ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
102052025-03-07T22:44:24.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102062025-03-07T22:44:24.031ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
102072025-03-07T22:44:24.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
102082025-03-07T22:44:24.032ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102092025-03-07T22:44:24.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102102025-03-07T22:44:24.033ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
102112025-03-07T22:44:24.033ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 63.198737ms iteration = 22 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
102122025-03-07T22:44:24.034ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102132025-03-07T22:44:24.035ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
102142025-03-07T22:44:24.035ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
102152025-03-07T22:44:24.035ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 65.037364ms iteration = 22 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
102162025-03-07T22:44:24.038ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
102172025-03-07T22:44:24.038ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
102182025-03-07T22:44:24.038ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102192025-03-07T22:44:24.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102202025-03-07T22:44:24.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
102212025-03-07T22:44:24.040ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 70.022593ms iteration = 22 status = 0
102222025-03-07T22:44:24.040ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102232025-03-07T22:44:24.042ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
102242025-03-07T22:44:24.042ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
102252025-03-07T22:44:24.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
102262025-03-07T22:44:24.043ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102272025-03-07T22:44:24.045ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
102282025-03-07T22:44:24.045ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
102292025-03-07T22:44:24.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
102302025-03-07T22:44:24.069ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 98.009062ms iteration = 22 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
102312025-03-07T22:44:24.074ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
102322025-03-07T22:44:24.074ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
102332025-03-07T22:44:24.075ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "77f49874-9ac8-4c6e-8beb-eb2516ca00a4", "content-length": "26", "date": "Fri, 07 Mar 2025 22:44:24 GMT"} })
102342025-03-07T22:44:24.075ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
102352025-03-07T22:44:24.078ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
102362025-03-07T22:44:24.092ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
102372025-03-07T22:44:24.092ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
102382025-03-07T22:44:24.093ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "53375a5f-589b-41a8-b958-2a0ec51b41d4", "content-length": "26", "date": "Fri, 07 Mar 2025 22:44:24 GMT"} })
102392025-03-07T22:44:24.093ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
102402025-03-07T22:44:24.095ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
102412025-03-07T22:44:24.158ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
102422025-03-07T22:44:24.158ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
102432025-03-07T22:44:24.158ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe292afa-5c99-42dd-b7a7-7a8bd115989b", "content-length": "26", "date": "Fri, 07 Mar 2025 22:44:24 GMT"} })
102442025-03-07T22:44:24.158ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
102452025-03-07T22:44:24.160ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
102462025-03-07T22:44:24.231ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 261.553643ms iteration = 22 status = {}
102472025-03-07T22:44:24.238ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
102482025-03-07T22:44:24.239ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "46c3f8b2-5e8a-4a13-aeab-57c608a097a3", "content-length": "691", "date": "Fri, 07 Mar 2025 22:44:24 GMT"} })
102492025-03-07T22:44:24.239ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
102502025-03-07T22:44:24.239ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
102512025-03-07T22:44:24.239ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
102522025-03-07T22:44:24.240ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 270.034735ms iteration = 22 status = {}
102532025-03-07T22:44:24.306ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
102542025-03-07T22:44:24.306ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5f14d74-5265-4fb9-acfc-c14ce1619eaa", "content-length": "1", "date": "Fri, 07 Mar 2025 22:44:24 GMT"} })
102552025-03-07T22:44:24.308ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
102562025-03-07T22:44:24.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
102572025-03-07T22:44:24.310ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 340.361901ms iteration = 22 status = null
102582025-03-07T22:44:24.312ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
102592025-03-07T22:44:24.312ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
102602025-03-07T22:44:24.313ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c01ce09-5aa0-43ca-ada9-983883182338", "content-length": "372", "date": "Fri, 07 Mar 2025 22:44:24 GMT"} })
102612025-03-07T22:44:24.313ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
102622025-03-07T22:44:24.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
102632025-03-07T22:44:24.334ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
102642025-03-07T22:44:24.334ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
102652025-03-07T22:44:24.334ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "09df0cab-e44b-4a97-b2e6-3674da21046b", "date": "Fri, 07 Mar 2025 22:44:24 GMT"} })
102662025-03-07T22:44:24.334ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
102672025-03-07T22:44:24.335ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
102682025-03-07T22:44:24.337ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
102692025-03-07T22:44:24.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
102702025-03-07T22:44:24.341ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
102712025-03-07T22:44:24.341ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
102722025-03-07T22:44:24.345ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
102732025-03-07T22:44:24.345ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
102742025-03-07T22:44:24.345ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "37cb92de-6f2e-43c1-b766-f1207b80d54e", "date": "Fri, 07 Mar 2025 22:44:24 GMT"} })
102752025-03-07T22:44:24.347ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 376.911132ms iteration = 22 status = {}
102762025-03-07T22:44:24.367ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
102772025-03-07T22:44:24.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
102782025-03-07T22:44:24.369ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
102792025-03-07T22:44:24.369ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
102802025-03-07T22:44:24.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
102812025-03-07T22:44:24.379ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
102822025-03-07T22:44:24.379ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff9131a3-aa92-4acd-bc89-4d2bcb1f7b91", "content-length": "134", "date": "Fri, 07 Mar 2025 22:44:24 GMT"} })
102832025-03-07T22:44:24.379ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
102842025-03-07T22:44:24.379ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
102852025-03-07T22:44:24.379ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
102862025-03-07T22:44:24.379ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
102872025-03-07T22:44:24.379ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
102882025-03-07T22:44:24.379ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
102892025-03-07T22:44:24.379ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
102902025-03-07T22:44:24.379ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
102912025-03-07T22:44:24.379ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
102922025-03-07T22:44:24.379ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
102932025-03-07T22:44:24.380ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
102942025-03-07T22:44:24.380ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
102952025-03-07T22:44:24.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
102962025-03-07T22:44:24.382ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
102972025-03-07T22:44:24.382ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "1106ed09-be39-40a3-9d0c-dfe08eb071c2", "content-length": "2", "date": "Fri, 07 Mar 2025 22:44:24 GMT"} })
102982025-03-07T22:44:24.382ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
102992025-03-07T22:44:24.382ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3664d9f-5c0d-4b5d-87d1-fc188a7498e7", "content-length": "52", "date": "Fri, 07 Mar 2025 22:44:24 GMT"} })
103002025-03-07T22:44:24.382ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
103012025-03-07T22:44:24.382ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
103022025-03-07T22:44:24.517ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "588b006f-d6b8-4185-9663-7c97cef92f66", "date": "Fri, 07 Mar 2025 22:44:24 GMT"} })
103032025-03-07T22:44:24.517ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
103042025-03-07T22:44:24.517ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
103052025-03-07T22:44:24.517ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c399ec81-1c27-422e-a54b-69bb27fe3c8e", "content-length": "682", "date": "Fri, 07 Mar 2025 22:44:24 GMT"} })
103062025-03-07T22:44:24.517ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103072025-03-07T22:44:24.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103082025-03-07T22:44:24.521ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103092025-03-07T22:44:24.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103102025-03-07T22:44:24.524ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103112025-03-07T22:44:24.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103122025-03-07T22:44:24.527ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103132025-03-07T22:44:24.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103142025-03-07T22:44:24.534ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 564.299753ms iteration = 22 status = {}
103152025-03-07T22:44:24.680ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 257 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 3f830050-eee5-40e6-ae5e-df1968179131 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
103162025-03-07T22:44:33.970ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 65 reason = Timeout
103172025-03-07T22:44:33.970ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 65 reason = Timeout
103182025-03-07T22:44:33.970ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
103192025-03-07T22:44:33.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
103202025-03-07T22:44:33.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
103212025-03-07T22:44:33.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
103222025-03-07T22:44:33.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103232025-03-07T22:44:33.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
103242025-03-07T22:44:33.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
103252025-03-07T22:44:33.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
103262025-03-07T22:44:33.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
103272025-03-07T22:44:33.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
103282025-03-07T22:44:33.978ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103292025-03-07T22:44:33.980ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
103302025-03-07T22:44:33.980ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
103312025-03-07T22:44:33.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
103322025-03-07T22:44:33.992ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
103332025-03-07T22:44:33.992ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.588877ms iteration = 65 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
103342025-03-07T22:44:34.000ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
103352025-03-07T22:44:34.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
103362025-03-07T22:44:34.036ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 66.224066ms iteration = 65 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
103372025-03-07T22:44:34.680ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 262 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = ffcdbffe-1f82-4ad6-aa98-9a337787e14a response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
103382025-03-07T22:44:43.970ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 66 reason = Timeout
103392025-03-07T22:44:43.970ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 66 reason = Timeout
103402025-03-07T22:44:43.970ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
103412025-03-07T22:44:43.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
103422025-03-07T22:44:43.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
103432025-03-07T22:44:43.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
103442025-03-07T22:44:43.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103452025-03-07T22:44:43.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
103462025-03-07T22:44:43.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
103472025-03-07T22:44:43.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
103482025-03-07T22:44:43.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
103492025-03-07T22:44:43.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
103502025-03-07T22:44:43.978ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103512025-03-07T22:44:43.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
103522025-03-07T22:44:43.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
103532025-03-07T22:44:43.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
103542025-03-07T22:44:43.992ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
103552025-03-07T22:44:43.992ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.575317ms iteration = 66 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
103562025-03-07T22:44:43.999ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
103572025-03-07T22:44:43.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
103582025-03-07T22:44:44.029ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 58.564923ms iteration = 66 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
103592025-03-07T22:44:44.680ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 241 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = a0e7e67f-8740-4f00-8569-f47bb7edf3a1 response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
103602025-03-07T22:44:53.970ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 67 reason = Timeout
103612025-03-07T22:44:53.970ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 25 reason = Timeout
103622025-03-07T22:44:53.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
103632025-03-07T22:44:53.970ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 12 reason = Timeout
103642025-03-07T22:44:53.970ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
103652025-03-07T22:44:53.970ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 23 reason = Timeout
103662025-03-07T22:44:53.970ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
103672025-03-07T22:44:53.970ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 23 reason = Timeout
103682025-03-07T22:44:53.970ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
103692025-03-07T22:44:53.970ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 23 reason = Timeout
103702025-03-07T22:44:53.970ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
103712025-03-07T22:44:53.970ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 13 reason = Timeout
103722025-03-07T22:44:53.970ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
103732025-03-07T22:44:53.970ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
103742025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 14 reason = Timeout
103752025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.22\u{b5}s iteration = 13 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
103762025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 12 reason = Timeout
103772025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 12 reason = Timeout
103782025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
103792025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 23 reason = Timeout
103802025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
103812025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
103822025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 23 reason = Timeout
103832025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 23 reason = Timeout
103842025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
103852025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 14 reason = Timeout
103862025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 12 reason = Timeout
103872025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 12 reason = Timeout
103882025-03-07T22:44:53.971ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
103892025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
103902025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 12 reason = Timeout
103912025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
103922025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
103932025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
103942025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 23 reason = Timeout
103952025-03-07T22:44:53.971ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
103962025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 23 reason = Timeout
103972025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
103982025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103992025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 23 reason = Timeout
104002025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 67 reason = Timeout
104012025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
104022025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
104032025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 12 reason = Timeout
104042025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
104052025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 23 reason = Timeout
104062025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
104072025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 12 reason = Timeout
104082025-03-07T22:44:53.971ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:34:53.971074957 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
104092025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
104102025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 23 reason = Timeout
104112025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
104122025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 23 reason = Timeout
104132025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
104142025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
104152025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 12 reason = Timeout
104162025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 23 reason = Timeout
104172025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104182025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
104192025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
104202025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 23 reason = Timeout
104212025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
104222025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 23 reason = Timeout
104232025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 12 reason = Timeout
104242025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
104252025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
104262025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
104272025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 23 reason = Timeout
104282025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
104292025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
104302025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
104312025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
104322025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
104332025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
104342025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
104352025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
104362025-03-07T22:44:53.971ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1452, tv_nsec: 318821443 } })
104372025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
104382025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
104392025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
104402025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
104412025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
104422025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
104432025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
104442025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
104452025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
104462025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
104472025-03-07T22:44:53.971ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1452, tv_nsec: 318904154 } })
104482025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
104492025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
104502025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
104512025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
104522025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
104532025-03-07T22:44:53.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
104542025-03-07T22:44:53.972ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
104552025-03-07T22:44:53.972ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.292193ms iteration = 12 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
104562025-03-07T22:44:53.972ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
104572025-03-07T22:44:53.972ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.567005ms iteration = 12 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
104582025-03-07T22:44:53.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104592025-03-07T22:44:53.973ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104602025-03-07T22:44:53.974ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.815836ms iteration = 23 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
104612025-03-07T22:44:53.975ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1452, tv_nsec: 322176615 } })
104622025-03-07T22:44:53.975ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104632025-03-07T22:44:53.975ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
104642025-03-07T22:44:53.976ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
104652025-03-07T22:44:53.976ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
104662025-03-07T22:44:53.976ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
104672025-03-07T22:44:53.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
104682025-03-07T22:44:53.976ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
104692025-03-07T22:44:53.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
104702025-03-07T22:44:53.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
104712025-03-07T22:44:53.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
104722025-03-07T22:44:53.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104732025-03-07T22:44:53.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104742025-03-07T22:44:53.977ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
104752025-03-07T22:44:53.977ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
104762025-03-07T22:44:53.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
104772025-03-07T22:44:53.978ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
104782025-03-07T22:44:53.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
104792025-03-07T22:44:53.978ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104802025-03-07T22:44:53.979ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
104812025-03-07T22:44:53.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
104822025-03-07T22:44:53.980ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
104832025-03-07T22:44:53.980ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
104842025-03-07T22:44:53.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
104852025-03-07T22:44:53.980ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1452, tv_nsec: 327931140 } })
104862025-03-07T22:44:53.981ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
104872025-03-07T22:44:53.981ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104882025-03-07T22:44:53.981ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104892025-03-07T22:44:53.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
104902025-03-07T22:44:53.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
104912025-03-07T22:44:53.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
104922025-03-07T22:44:53.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104932025-03-07T22:44:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
104942025-03-07T22:44:53.983ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
104952025-03-07T22:44:53.983ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
104962025-03-07T22:44:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
104972025-03-07T22:44:53.983ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
104982025-03-07T22:44:53.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
104992025-03-07T22:44:53.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
105002025-03-07T22:44:53.985ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
105012025-03-07T22:44:53.985ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
105022025-03-07T22:44:53.985ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105032025-03-07T22:44:53.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
105042025-03-07T22:44:53.985ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:34:53.971074957 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
105052025-03-07T22:44:53.985ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 14.620661ms iteration = 12 status = {"expiration":"2025-03-07T22:34:53.971074957Z","pruned":[]}
105062025-03-07T22:44:53.987ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105072025-03-07T22:44:53.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
105082025-03-07T22:44:53.987ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
105092025-03-07T22:44:53.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
105102025-03-07T22:44:53.988ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
105112025-03-07T22:44:53.988ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
105122025-03-07T22:44:53.988ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 16.975243ms iteration = 12 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
105132025-03-07T22:44:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
105142025-03-07T22:44:53.989ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
105152025-03-07T22:44:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
105162025-03-07T22:44:53.991ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 20.457057ms iteration = 12 status = {"errors":[],"found_port_ok":[]}
105172025-03-07T22:44:53.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
105182025-03-07T22:44:53.992ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
105192025-03-07T22:44:53.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
105202025-03-07T22:44:53.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
105212025-03-07T22:44:53.993ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105222025-03-07T22:44:53.995ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105232025-03-07T22:44:53.995ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1452, tv_nsec: 342800133 } })
105242025-03-07T22:44:53.995ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
105252025-03-07T22:44:53.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
105262025-03-07T22:44:53.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
105272025-03-07T22:44:54.002ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
105282025-03-07T22:44:54.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
105292025-03-07T22:44:54.004ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
105302025-03-07T22:44:54.004ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
105312025-03-07T22:44:54.004ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
105322025-03-07T22:44:54.004ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
105332025-03-07T22:44:54.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
105342025-03-07T22:44:54.005ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
105352025-03-07T22:44:54.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
105362025-03-07T22:44:54.007ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
105372025-03-07T22:44:54.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
105382025-03-07T22:44:54.007ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105392025-03-07T22:44:54.008ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105402025-03-07T22:44:54.010ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
105412025-03-07T22:44:54.010ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
105422025-03-07T22:44:54.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
105432025-03-07T22:44:54.013ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
105442025-03-07T22:44:54.013ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
105452025-03-07T22:44:54.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
105462025-03-07T22:44:54.015ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105472025-03-07T22:44:54.022ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
105482025-03-07T22:44:54.022ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
105492025-03-07T22:44:54.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
105502025-03-07T22:44:54.026ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
105512025-03-07T22:44:54.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
105522025-03-07T22:44:54.032ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 61.023146ms iteration = 23 status = {"errors":[],"finish_invoked_ok":[]}
105532025-03-07T22:44:54.039ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 68.317127ms iteration = 23 status = {"errors":[],"garbage_collect_requested":[]}
105542025-03-07T22:44:54.040ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
105552025-03-07T22:44:54.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
105562025-03-07T22:44:54.042ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 70.883071ms iteration = 23 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
105572025-03-07T22:44:54.042ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 71.566077ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
105582025-03-07T22:44:54.043ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
105592025-03-07T22:44:54.043ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
105602025-03-07T22:44:54.043ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
105612025-03-07T22:44:54.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
105622025-03-07T22:44:54.045ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 74.745298ms iteration = 67 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
105632025-03-07T22:44:54.046ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
105642025-03-07T22:44:54.046ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
105652025-03-07T22:44:54.046ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 76.131681ms iteration = 23 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
105662025-03-07T22:44:54.053ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
105672025-03-07T22:44:54.053ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
105682025-03-07T22:44:54.053ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
105692025-03-07T22:44:54.053ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
105702025-03-07T22:44:54.053ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
105712025-03-07T22:44:54.054ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105722025-03-07T22:44:54.056ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 84.723894ms iteration = 23 status = {"errors":[],"requests_created_ok":[]}
105732025-03-07T22:44:54.056ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
105742025-03-07T22:44:54.056ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
105752025-03-07T22:44:54.056ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 85.637893ms iteration = 12 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
105762025-03-07T22:44:54.056ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105772025-03-07T22:44:54.058ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
105782025-03-07T22:44:54.058ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
105792025-03-07T22:44:54.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
105802025-03-07T22:44:54.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
105812025-03-07T22:44:54.058ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
105822025-03-07T22:44:54.058ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
105832025-03-07T22:44:54.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
105842025-03-07T22:44:54.059ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
105852025-03-07T22:44:54.059ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
105862025-03-07T22:44:54.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
105872025-03-07T22:44:54.060ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105882025-03-07T22:44:54.061ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105892025-03-07T22:44:54.062ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
105902025-03-07T22:44:54.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
105912025-03-07T22:44:54.062ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
105922025-03-07T22:44:54.062ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
105932025-03-07T22:44:54.062ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
105942025-03-07T22:44:54.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
105952025-03-07T22:44:54.063ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
105962025-03-07T22:44:54.063ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 92.606839ms iteration = 23 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
105972025-03-07T22:44:54.063ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
105982025-03-07T22:44:54.063ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
105992025-03-07T22:44:54.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
106002025-03-07T22:44:54.064ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
106012025-03-07T22:44:54.064ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
106022025-03-07T22:44:54.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
106032025-03-07T22:44:54.068ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106042025-03-07T22:44:54.068ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106052025-03-07T22:44:54.073ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
106062025-03-07T22:44:54.073ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
106072025-03-07T22:44:54.073ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
106082025-03-07T22:44:54.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
106092025-03-07T22:44:54.073ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
106102025-03-07T22:44:54.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
106112025-03-07T22:44:54.074ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
106122025-03-07T22:44:54.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
106132025-03-07T22:44:54.077ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
106142025-03-07T22:44:54.077ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
106152025-03-07T22:44:54.077ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
106162025-03-07T22:44:54.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
106172025-03-07T22:44:54.082ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106182025-03-07T22:44:54.087ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 116.143536ms iteration = 23 status = 0
106192025-03-07T22:44:54.087ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
106202025-03-07T22:44:54.087ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
106212025-03-07T22:44:54.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
106222025-03-07T22:44:54.095ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
106232025-03-07T22:44:54.095ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
106242025-03-07T22:44:54.095ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-07 22:34:17.208516 UTC dns_group = internal generation = 1
106252025-03-07T22:44:54.095ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 124.322855ms iteration = 12 status = {"generation":1}
106262025-03-07T22:44:54.097ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
106272025-03-07T22:44:54.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
106282025-03-07T22:44:54.099ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106292025-03-07T22:44:54.102ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
106302025-03-07T22:44:54.102ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
106312025-03-07T22:44:54.102ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 132.057309ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
106322025-03-07T22:44:54.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
106332025-03-07T22:44:54.104ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
106342025-03-07T22:44:54.104ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
106352025-03-07T22:44:54.104ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-07 22:34:17.797135 UTC dns_group = external generation = 2
106362025-03-07T22:44:54.104ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 133.648974ms iteration = 12 status = {"generation":2}
106372025-03-07T22:44:54.106ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
106382025-03-07T22:44:54.108ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
106392025-03-07T22:44:54.108ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
106402025-03-07T22:44:54.108ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1734f0d-d9f7-421f-a9cf-7609e76d6e65", "content-length": "26", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} })
106412025-03-07T22:44:54.108ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
106422025-03-07T22:44:54.111ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
106432025-03-07T22:44:54.112ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
106442025-03-07T22:44:54.112ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 141.675331ms iteration = 67 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
106452025-03-07T22:44:54.126ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
106462025-03-07T22:44:54.126ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 155.486424ms iteration = 23 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
106472025-03-07T22:44:54.140ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
106482025-03-07T22:44:54.146ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
106492025-03-07T22:44:54.146ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
106502025-03-07T22:44:54.147ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b651ebe-054d-4641-80c7-96bba79c48be", "content-length": "26", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} })
106512025-03-07T22:44:54.147ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
106522025-03-07T22:44:54.149ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
106532025-03-07T22:44:54.151ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) }
106542025-03-07T22:44:54.151ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 180.199601ms iteration = 12 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
106552025-03-07T22:44:54.152ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
106562025-03-07T22:44:54.152ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
106572025-03-07T22:44:54.152ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c336c5d-022a-48af-aa0a-6d6a9cb6dbd7", "content-length": "26", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} })
106582025-03-07T22:44:54.152ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
106592025-03-07T22:44:54.154ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
106602025-03-07T22:44:54.154ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 183.404232ms iteration = 25 status = {"physical_disks_added":0}
106612025-03-07T22:44:54.158ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
106622025-03-07T22:44:54.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
106632025-03-07T22:44:54.165ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
106642025-03-07T22:44:54.166ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
106652025-03-07T22:44:54.167ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106662025-03-07T22:44:54.169ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
106672025-03-07T22:44:54.169ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
106682025-03-07T22:44:54.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
106692025-03-07T22:44:54.171ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106702025-03-07T22:44:54.173ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
106712025-03-07T22:44:54.173ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
106722025-03-07T22:44:54.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
106732025-03-07T22:44:54.198ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
106742025-03-07T22:44:54.198ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 227.951531ms iteration = 12 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
106752025-03-07T22:44:54.280ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
106762025-03-07T22:44:54.280ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "7229f5c6-29cd-48ba-aa10-3dc9a33094c1", "content-length": "1", "date": "Fri, 07 Mar 2025 22:44:53 GMT"} })
106772025-03-07T22:44:54.285ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
106782025-03-07T22:44:54.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
106792025-03-07T22:44:54.288ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 317.256809ms iteration = 23 status = null
106802025-03-07T22:44:54.289ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
106812025-03-07T22:44:54.327ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
106822025-03-07T22:44:54.327ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "f35ca3c9-61dc-433c-b762-1aa401fd1363", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
106832025-03-07T22:44:54.327ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
106842025-03-07T22:44:54.330ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "af7ed0fe-ccc2-4710-b2dc-d8d4c5dfacc4", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
106852025-03-07T22:44:54.330ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
106862025-03-07T22:44:54.332ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 361.316472ms iteration = 23 status = {}
106872025-03-07T22:44:54.333ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
106882025-03-07T22:44:54.333ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 362.15789ms iteration = 14 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
106892025-03-07T22:44:54.408ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
106902025-03-07T22:44:54.408ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e98a9d7-d2b3-4389-98c2-a3fe7f55c4c8", "content-length": "691", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} })
106912025-03-07T22:44:54.408ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
106922025-03-07T22:44:54.408ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
106932025-03-07T22:44:54.408ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
106942025-03-07T22:44:54.410ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 439.08102ms iteration = 23 status = {}
106952025-03-07T22:44:54.429ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
106962025-03-07T22:44:54.430ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "b571743e-187a-4df6-9a6b-152606a93540", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
106972025-03-07T22:44:54.430ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
106982025-03-07T22:44:54.432ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "341e57da-91c0-42b5-b2e1-1df1981cb605", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
106992025-03-07T22:44:54.432ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
107002025-03-07T22:44:54.434ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "32c26d1d-a154-4d12-b349-b6615b4e4beb", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
107012025-03-07T22:44:54.434ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
107022025-03-07T22:44:54.437ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
107032025-03-07T22:44:54.437ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 466.625134ms iteration = 14 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
107042025-03-07T22:44:54.451ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
107052025-03-07T22:44:54.451ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
107062025-03-07T22:44:54.451ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f185575-1b9d-492c-a288-1a14a9ec3f77", "content-length": "372", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} })
107072025-03-07T22:44:54.451ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
107082025-03-07T22:44:54.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
107092025-03-07T22:44:54.472ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
107102025-03-07T22:44:54.472ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
107112025-03-07T22:44:54.472ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "4718b290-1166-4e32-bc51-029faf86b6c5", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} })
107122025-03-07T22:44:54.472ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
107132025-03-07T22:44:54.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
107142025-03-07T22:44:54.475ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
107152025-03-07T22:44:54.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
107162025-03-07T22:44:54.479ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
107172025-03-07T22:44:54.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
107182025-03-07T22:44:54.485ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
107192025-03-07T22:44:54.485ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
107202025-03-07T22:44:54.486ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "66b65733-a176-4eaa-b2b8-15468985933c", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} })
107212025-03-07T22:44:54.489ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 518.294061ms iteration = 23 status = {}
107222025-03-07T22:44:54.548ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
107232025-03-07T22:44:54.548ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
107242025-03-07T22:44:54.550ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
107252025-03-07T22:44:54.550ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
107262025-03-07T22:44:54.551ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
107272025-03-07T22:44:54.561ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
107282025-03-07T22:44:54.562ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c0a2d5b-9241-475a-a90f-29b177b5a4a4", "content-length": "134", "date": "Fri, 07 Mar 2025 22:44:53 GMT"} })
107292025-03-07T22:44:54.562ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
107302025-03-07T22:44:54.562ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
107312025-03-07T22:44:54.562ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
107322025-03-07T22:44:54.562ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
107332025-03-07T22:44:54.562ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
107342025-03-07T22:44:54.562ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
107352025-03-07T22:44:54.562ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
107362025-03-07T22:44:54.562ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
107372025-03-07T22:44:54.562ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
107382025-03-07T22:44:54.562ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
107392025-03-07T22:44:54.562ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
107402025-03-07T22:44:54.562ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
107412025-03-07T22:44:54.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
107422025-03-07T22:44:54.564ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
107432025-03-07T22:44:54.565ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "50c74c16-4346-4c96-89c4-3ed20375f475", "content-length": "2", "date": "Fri, 07 Mar 2025 22:44:53 GMT"} })
107442025-03-07T22:44:54.565ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
107452025-03-07T22:44:54.565ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbfb094e-7511-4567-a1e0-3992c1217edc", "content-length": "52", "date": "Fri, 07 Mar 2025 22:44:53 GMT"} })
107462025-03-07T22:44:54.565ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
107472025-03-07T22:44:54.565ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
107482025-03-07T22:44:54.681ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 464 local_addr = [fd00:1122:3344:101::b]:53580 method = GET remote_addr = [fd00:1122:3344:101::d]:42651 req_id = 779d4c73-b766-466f-bb8c-efcd36b29b7d response_code = 200 uri = /dfe169a3-7536-4b54-9593-8815d6e81dd4
107492025-03-07T22:44:54.766ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "bfa72272-c01a-492c-b65e-274d5bca2c19", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} })
107502025-03-07T22:44:54.766ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
107512025-03-07T22:44:54.766ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
107522025-03-07T22:44:54.766ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "e361895f-7790-4a53-8698-77bc9281809d", "content-length": "682", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} })
107532025-03-07T22:44:54.767ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
107542025-03-07T22:44:54.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
107552025-03-07T22:44:54.771ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
107562025-03-07T22:44:54.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
107572025-03-07T22:44:54.774ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
107582025-03-07T22:44:54.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
107592025-03-07T22:44:54.775ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
107602025-03-07T22:44:54.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
107612025-03-07T22:44:54.783ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 812.461107ms iteration = 23 status = {}
107622025-03-07T22:44:54.814ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
107632025-03-07T22:44:54.814ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
107642025-03-07T22:44:54.845ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
107652025-03-07T22:44:54.845ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:35280
107662025-03-07T22:44:54.845ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:35280 req_id = 7fbf1691-ed17-4f74-a0be-73502b5b310c roles = RoleSet { roles: {} } uri = /metrics/producers
107672025-03-07T22:44:54.846ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:35280 req_id = 7fbf1691-ed17-4f74-a0be-73502b5b310c resource = Database result = Ok(()) uri = /metrics/producers
107682025-03-07T22:44:54.881ZINFOnexus: assigned collector to new producer collector_id = 7b5e9a24-6b39-4a58-a275-8cc5a75195d8 file = nexus/src/app/oximeter.rs:76 producer_id = dfe169a3-7536-4b54-9593-8815d6e81dd4
107692025-03-07T22:44:54.881ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 35804 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:35280 req_id = 7fbf1691-ed17-4f74-a0be-73502b5b310c response_code = 201 uri = /metrics/producers
107702025-03-07T22:44:54.881ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "7fbf1691-ed17-4f74-a0be-73502b5b310c", "content-length": "41", "date": "Fri, 07 Mar 2025 22:44:54 GMT"} })
107712025-03-07T22:44:54.883ZDEBGnexus (producer-registration-task): registered with nexus successfully
107722025-03-07T22:44:54.883ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s