Max level shown:
12025-05-30T21:30:05.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 24 reason = Timeout
22025-05-30T21:30:05.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32025-05-30T21:30:05.939ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 24 reason = Timeout
42025-05-30T21:30:05.939ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52025-05-30T21:30:05.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62025-05-30T21:30:05.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72025-05-30T21:30:05.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82025-05-30T21:30:05.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92025-05-30T21:30:05.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102025-05-30T21:30:05.943ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.822694ms iteration = 24 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
112025-05-30T21:30:05.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
122025-05-30T21:30:05.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
132025-05-30T21:30:05.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
142025-05-30T21:30:05.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
152025-05-30T21:30:05.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
162025-05-30T21:30:05.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
172025-05-30T21:30:05.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
182025-05-30T21:30:05.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
192025-05-30T21:30:05.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
202025-05-30T21:30:05.962ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
212025-05-30T21:30:05.962ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.298368ms iteration = 24 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
222025-05-30T21:30:08.774ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 298 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 40ff5671-0021-43e3-9085-5fe420bb7de7 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
232025-05-30T21:30:15.937ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 5 reason = Timeout
242025-05-30T21:30:15.937ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
252025-05-30T21:30:15.937ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
262025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
272025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 25 reason = Timeout
282025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
292025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
302025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 9 reason = Timeout
312025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 5 reason = Timeout
322025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
332025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
342025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 7 reason = Timeout
352025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 7 reason = Timeout
362025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 9 reason = Timeout
372025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 5 reason = Timeout
382025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
392025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
402025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 9 reason = Timeout
412025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
422025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 9 reason = Timeout
432025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 9 reason = Timeout
442025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
452025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
462025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 5 reason = Timeout
472025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 9 reason = Timeout
482025-05-30T21:30:15.938ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
492025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 9 reason = Timeout
502025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
512025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
522025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
532025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 9 reason = Timeout
542025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 9 reason = Timeout
552025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
562025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
572025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
582025-05-30T21:30:15.938ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 10 reason = Timeout
592025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 9 reason = Timeout
602025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
612025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
622025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 5 reason = Timeout
632025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 5 reason = Timeout
642025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
652025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
662025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 9 reason = Timeout
672025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
682025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 9 reason = Timeout
692025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
702025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 9 reason = Timeout
712025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
722025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 5 reason = Timeout
732025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
742025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 9 reason = Timeout
752025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
762025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
772025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 5 reason = Timeout
782025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 9 reason = Timeout
792025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
802025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 25 reason = Timeout
812025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
822025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 6 reason = Timeout
832025-05-30T21:30:15.939ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e
842025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.89\u{b5}s iteration = 6 status = {"enabled":false,"target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e"}
852025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 5 reason = Timeout
862025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 5 reason = Timeout
872025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
882025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
892025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 5 reason = Timeout
902025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 9 reason = Timeout
912025-05-30T21:30:15.939ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
922025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
932025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 5 reason = Timeout
942025-05-30T21:30:15.939ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-30 21:20:15.939163558 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
952025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
962025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
972025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
982025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
992025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
1002025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1012025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1022025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
1032025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1042025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1052025-05-30T21:30:15.939ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2a161087-f464-41f8-b020-0f9e83fd5351.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1350, tv_nsec: 878757592 } })
1062025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
1072025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1082025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1092025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
1102025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1112025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
1122025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
1132025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1142025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
1152025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
1162025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1172025-05-30T21:30:15.939ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0413f6b4-86a1-42a1-919a-e8f42ff9f638.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("acb1452d-0a02-4b36-a462-c8ce2f96a488.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:2::1))) }], valid_until: Instant { tv_sec: 1350, tv_nsec: 878900703 } })
1182025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
1192025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1202025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1212025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1222025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1232025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
1242025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
1252025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
1262025-05-30T21:30:15.939ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 908.18\u{b5}s iteration = 5 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
1272025-05-30T21:30:15.940ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } dns_group = internal
1282025-05-30T21:30:15.940ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 993.961\u{b5}s iteration = 5 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353"]}
1292025-05-30T21:30:15.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1302025-05-30T21:30:15.941ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.960074ms iteration = 9 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
1312025-05-30T21:30:15.941ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1322025-05-30T21:30:15.943ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1332025-05-30T21:30:15.944ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
1342025-05-30T21:30:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1352025-05-30T21:30:15.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1362025-05-30T21:30:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1372025-05-30T21:30:15.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1382025-05-30T21:30:15.945ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1392025-05-30T21:30:15.945ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
1402025-05-30T21:30:15.945ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1412025-05-30T21:30:15.945ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1422025-05-30T21:30:15.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1432025-05-30T21:30:15.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1442025-05-30T21:30:15.947ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 8.509928ms iteration = 9 status = {"errors":[],"finish_invoked_ok":[]}
1452025-05-30T21:30:15.948ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 9.61042ms iteration = 9 status = {"errors":[],"garbage_collect_requested":[]}
1462025-05-30T21:30:15.948ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1472025-05-30T21:30:15.949ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1482025-05-30T21:30:15.949ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1492025-05-30T21:30:15.950ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1350, tv_nsec: 889898100 } })
1502025-05-30T21:30:15.950ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
1512025-05-30T21:30:15.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1522025-05-30T21:30:15.951ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1532025-05-30T21:30:15.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1542025-05-30T21:30:15.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
1552025-05-30T21:30:15.951ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
1562025-05-30T21:30:15.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1572025-05-30T21:30:15.952ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
1582025-05-30T21:30:15.952ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1592025-05-30T21:30:15.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1602025-05-30T21:30:15.954ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1612025-05-30T21:30:15.954ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1622025-05-30T21:30:15.955ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 16.795413ms iteration = 9 status = {"errors":[],"requests_created_ok":[]}
1632025-05-30T21:30:15.955ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 16.881853ms iteration = 5 status = {"errors":[],"found_port_ok":[]}
1642025-05-30T21:30:15.956ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1350, tv_nsec: 895433203 } })
1652025-05-30T21:30:15.957ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
1662025-05-30T21:30:15.957ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
1672025-05-30T21:30:15.957ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
1682025-05-30T21:30:15.957ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
1692025-05-30T21:30:15.957ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
1702025-05-30T21:30:15.957ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1712025-05-30T21:30:15.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1722025-05-30T21:30:15.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1732025-05-30T21:30:15.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1742025-05-30T21:30:15.961ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1752025-05-30T21:30:15.961ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1762025-05-30T21:30:15.963ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
1772025-05-30T21:30:15.963ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1782025-05-30T21:30:15.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1792025-05-30T21:30:15.964ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1802025-05-30T21:30:15.964ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1812025-05-30T21:30:15.964ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:202 num_receivers = 0
1822025-05-30T21:30:15.964ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 25.204969ms iteration = 5 status = {"by_rx":{},"error":null}
1832025-05-30T21:30:15.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1842025-05-30T21:30:15.964ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-30 21:20:15.939163558 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
1852025-05-30T21:30:15.964ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
1862025-05-30T21:30:15.964ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
1872025-05-30T21:30:15.964ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 25.295461ms iteration = 5 status = {"expiration":"2025-05-30T21:20:15.939163558Z","pruned":[]}
1882025-05-30T21:30:15.964ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 25.350431ms iteration = 5 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
1892025-05-30T21:30:15.966ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
1902025-05-30T21:30:15.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1912025-05-30T21:30:15.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
1922025-05-30T21:30:15.966ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1932025-05-30T21:30:15.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1942025-05-30T21:30:15.967ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1952025-05-30T21:30:15.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1962025-05-30T21:30:15.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
1972025-05-30T21:30:15.971ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1982025-05-30T21:30:15.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1992025-05-30T21:30:15.971ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2002025-05-30T21:30:15.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2012025-05-30T21:30:15.972ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2022025-05-30T21:30:15.972ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.730388ms iteration = 25 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
2032025-05-30T21:30:15.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2042025-05-30T21:30:15.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2052025-05-30T21:30:15.974ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2062025-05-30T21:30:15.976ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2072025-05-30T21:30:15.977ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
2082025-05-30T21:30:15.977ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2092025-05-30T21:30:15.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2102025-05-30T21:30:15.978ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 39.326652ms iteration = 5 status = {"dispatched":[],"errors":[],"glob_version":"144.0.0","globs_reprocessed":{},"no_receivers":[]}
2112025-05-30T21:30:15.979ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1350, tv_nsec: 918155084 } })
2122025-05-30T21:30:15.979ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
2132025-05-30T21:30:15.980ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
2142025-05-30T21:30:15.980ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2152025-05-30T21:30:15.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2162025-05-30T21:30:15.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
2172025-05-30T21:30:15.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2182025-05-30T21:30:15.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2192025-05-30T21:30:15.983ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2202025-05-30T21:30:15.989ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
2212025-05-30T21:30:15.989ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2222025-05-30T21:30:15.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2232025-05-30T21:30:15.993ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
2242025-05-30T21:30:15.993ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
2252025-05-30T21:30:15.993ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
2262025-05-30T21:30:15.993ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
2272025-05-30T21:30:15.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
2282025-05-30T21:30:15.996ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
2292025-05-30T21:30:15.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
2302025-05-30T21:30:15.998ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
2312025-05-30T21:30:15.998ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 60.13864ms iteration = 25 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
2322025-05-30T21:30:16.004ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
2332025-05-30T21:30:16.004ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
2342025-05-30T21:30:16.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
2352025-05-30T21:30:16.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
2362025-05-30T21:30:16.007ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2372025-05-30T21:30:16.007ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
2382025-05-30T21:30:16.007ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
2392025-05-30T21:30:16.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
2402025-05-30T21:30:16.016ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
2412025-05-30T21:30:16.016ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
2422025-05-30T21:30:16.016ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
2432025-05-30T21:30:16.016ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
2442025-05-30T21:30:16.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
2452025-05-30T21:30:16.017ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2462025-05-30T21:30:16.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2472025-05-30T21:30:16.018ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 80.03394ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2482025-05-30T21:30:16.021ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
2492025-05-30T21:30:16.021ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 82.400606ms iteration = 9 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
2502025-05-30T21:30:16.026ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
2512025-05-30T21:30:16.026ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
2522025-05-30T21:30:16.026ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 87.243452ms iteration = 5 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
2532025-05-30T21:30:16.046ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
2542025-05-30T21:30:16.110ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
2552025-05-30T21:30:16.110ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2562025-05-30T21:30:16.111ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffcb57a2-8951-463a-b745-33e54463007e", "content-length": "26", "date": "Fri, 30 May 2025 21:30:16 GMT"} })
2572025-05-30T21:30:16.111ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2582025-05-30T21:30:16.112ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
2592025-05-30T21:30:16.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
2602025-05-30T21:30:16.113ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
2612025-05-30T21:30:16.136ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2622025-05-30T21:30:16.137ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2632025-05-30T21:30:16.163ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 224.328527ms iteration = 10 status = {"physical_disks_added":0}
2642025-05-30T21:30:16.165ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
2652025-05-30T21:30:16.165ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2662025-05-30T21:30:16.166ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "73ea3f05-a76a-4006-b570-42d323649641", "content-length": "26", "date": "Fri, 30 May 2025 21:30:16 GMT"} })
2672025-05-30T21:30:16.166ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2682025-05-30T21:30:16.168ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
2692025-05-30T21:30:16.170ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
2702025-05-30T21:30:16.170ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2712025-05-30T21:30:16.171ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "257667a8-2b90-413b-986f-ec8d0506131d", "content-length": "372", "date": "Fri, 30 May 2025 21:30:15 GMT"} })
2722025-05-30T21:30:16.171ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2732025-05-30T21:30:16.171ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2742025-05-30T21:30:16.175ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2752025-05-30T21:30:16.176ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 237.381037ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2762025-05-30T21:30:16.177ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
2772025-05-30T21:30:16.178ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2782025-05-30T21:30:16.179ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
2792025-05-30T21:30:16.179ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
2802025-05-30T21:30:16.179ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2812025-05-30T21:30:16.179ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
2822025-05-30T21:30:16.181ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2832025-05-30T21:30:16.184ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
2842025-05-30T21:30:16.184ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2852025-05-30T21:30:16.184ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2862025-05-30T21:30:16.186ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 248.002898ms iteration = 9 status = 0
2872025-05-30T21:30:16.195ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
2882025-05-30T21:30:16.195ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
2892025-05-30T21:30:16.196ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "93c3e2fb-f532-4501-88bc-b7c248ef8fc5", "date": "Fri, 30 May 2025 21:30:15 GMT"} })
2902025-05-30T21:30:16.196ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2912025-05-30T21:30:16.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2922025-05-30T21:30:16.199ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2932025-05-30T21:30:16.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2942025-05-30T21:30:16.206ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2952025-05-30T21:30:16.207ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2962025-05-30T21:30:16.211ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
2972025-05-30T21:30:16.211ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2982025-05-30T21:30:16.211ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "1a58bc44-0d9f-4577-bb3c-346fcbaf8911", "date": "Fri, 30 May 2025 21:30:15 GMT"} })
2992025-05-30T21:30:16.213ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 274.081948ms iteration = 9 status = {}
3002025-05-30T21:30:16.221ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
3012025-05-30T21:30:16.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
3022025-05-30T21:30:16.222ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3032025-05-30T21:30:16.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3042025-05-30T21:30:16.230ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3052025-05-30T21:30:16.230ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3062025-05-30T21:30:16.232ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3072025-05-30T21:30:16.233ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
3082025-05-30T21:30:16.233ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
3092025-05-30T21:30:16.233ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
3102025-05-30T21:30:16.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
3112025-05-30T21:30:16.233ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
3122025-05-30T21:30:16.233ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3132025-05-30T21:30:16.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3142025-05-30T21:30:16.235ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3152025-05-30T21:30:16.235ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3162025-05-30T21:30:16.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3172025-05-30T21:30:16.235ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
3182025-05-30T21:30:16.235ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
3192025-05-30T21:30:16.235ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca75f4cf-928c-455a-a9c0-27223075b659", "content-length": "26", "date": "Fri, 30 May 2025 21:30:16 GMT"} })
3202025-05-30T21:30:16.235ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
3212025-05-30T21:30:16.236ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3222025-05-30T21:30:16.236ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3232025-05-30T21:30:16.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3242025-05-30T21:30:16.237ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
3252025-05-30T21:30:16.238ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
3262025-05-30T21:30:16.238ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "760f4064-098c-4f80-9048-e58a3215f08e", "date": "Fri, 30 May 2025 21:30:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3272025-05-30T21:30:16.238ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3282025-05-30T21:30:16.240ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "f01a1910-4b49-4111-96d7-770e11583d3d", "date": "Fri, 30 May 2025 21:30:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3292025-05-30T21:30:16.240ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3302025-05-30T21:30:16.242ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3312025-05-30T21:30:16.242ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 303.683197ms iteration = 7 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
3322025-05-30T21:30:16.245ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
3332025-05-30T21:30:16.245ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 306.110126ms iteration = 9 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
3342025-05-30T21:30:16.249ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 310.614898ms iteration = 9 status = {}
3352025-05-30T21:30:16.264ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3362025-05-30T21:30:16.266ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
3372025-05-30T21:30:16.266ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3382025-05-30T21:30:16.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3392025-05-30T21:30:16.275ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 336.809379ms iteration = 9 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
3402025-05-30T21:30:16.275ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
3412025-05-30T21:30:16.275ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
3422025-05-30T21:30:16.275ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 336.93895ms iteration = 9 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
3432025-05-30T21:30:16.276ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
3442025-05-30T21:30:16.276ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 0 version = 2
3452025-05-30T21:30:16.276ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3462025-05-30T21:30:16.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3472025-05-30T21:30:16.278ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3482025-05-30T21:30:16.281ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
3492025-05-30T21:30:16.281ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
3502025-05-30T21:30:16.281ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3512025-05-30T21:30:16.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3522025-05-30T21:30:16.282ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3532025-05-30T21:30:16.282ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3542025-05-30T21:30:16.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3552025-05-30T21:30:16.284ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 1 version = 2
3562025-05-30T21:30:16.284ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
3572025-05-30T21:30:16.284ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-30 21:26:18.957109 UTC dns_group = external generation = 2
3582025-05-30T21:30:16.284ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 345.747351ms iteration = 5 status = {"generation":2}
3592025-05-30T21:30:16.288ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3602025-05-30T21:30:16.290ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
3612025-05-30T21:30:16.290ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3622025-05-30T21:30:16.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3632025-05-30T21:30:16.325ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3642025-05-30T21:30:16.329ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-30T21:26:13.986396Z, time_modified: 2025-05-30T21:26:13.986396Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }) }
3652025-05-30T21:30:16.329ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 390.534305ms iteration = 5 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]}},"default_endpoint":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
3662025-05-30T21:30:16.330ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3672025-05-30T21:30:16.330ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3682025-05-30T21:30:16.330ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3692025-05-30T21:30:16.333ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
3702025-05-30T21:30:16.333ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
3712025-05-30T21:30:16.333ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-30 21:26:17.226115 UTC dns_group = internal generation = 1
3722025-05-30T21:30:16.333ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 395.750316ms iteration = 5 status = {"generation":1}
3732025-05-30T21:30:16.336ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
3742025-05-30T21:30:16.336ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3752025-05-30T21:30:16.336ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3762025-05-30T21:30:16.339ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 400.047645ms iteration = 9 status = null
3772025-05-30T21:30:16.370ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
3782025-05-30T21:30:16.370ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 431.283413ms iteration = 5 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
3792025-05-30T21:30:16.379ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
3802025-05-30T21:30:16.379ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "12accf83-0a31-4dd9-96d0-1d879312e48c", "content-length": "691", "date": "Fri, 30 May 2025 21:30:15 GMT"} })
3812025-05-30T21:30:16.379ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
3822025-05-30T21:30:16.379ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
3832025-05-30T21:30:16.379ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
3842025-05-30T21:30:16.381ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 442.220969ms iteration = 9 status = {}
3852025-05-30T21:30:16.401ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
3862025-05-30T21:30:16.401ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "15ffc290-bddf-48ed-9873-5b1352b7a400", "date": "Fri, 30 May 2025 21:30:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
3872025-05-30T21:30:16.401ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
3882025-05-30T21:30:16.403ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "2d5a5ede-ebc3-4eb8-8b85-327f65dc2ec9", "date": "Fri, 30 May 2025 21:30:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
3892025-05-30T21:30:16.403ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
3902025-05-30T21:30:16.405ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "35e858ee-7f6b-4f64-ac06-001ad1f6559b", "date": "Fri, 30 May 2025 21:30:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
3912025-05-30T21:30:16.405ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
3922025-05-30T21:30:16.406ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
3932025-05-30T21:30:16.407ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 468.223628ms iteration = 7 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
3942025-05-30T21:30:16.411ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3952025-05-30T21:30:16.411ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3962025-05-30T21:30:16.413ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
3972025-05-30T21:30:16.413ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3982025-05-30T21:30:16.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3992025-05-30T21:30:16.425ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
4002025-05-30T21:30:16.425ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
4012025-05-30T21:30:16.425ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
4022025-05-30T21:30:16.425ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
4032025-05-30T21:30:16.425ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
4042025-05-30T21:30:16.425ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
4052025-05-30T21:30:16.425ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
4062025-05-30T21:30:16.425ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
4072025-05-30T21:30:16.426ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
4082025-05-30T21:30:16.426ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
4092025-05-30T21:30:16.426ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
4102025-05-30T21:30:16.426ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4112025-05-30T21:30:16.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4122025-05-30T21:30:16.428ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
4132025-05-30T21:30:16.428ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
4142025-05-30T21:30:16.531ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "b9d101f8-e1d3-4738-8dfa-28c47026da7d", "date": "Fri, 30 May 2025 21:30:15 GMT"} })
4152025-05-30T21:30:16.531ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
4162025-05-30T21:30:16.531ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
4172025-05-30T21:30:16.531ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ae95f73-027f-465f-9493-fe7bf2035c71", "content-length": "682", "date": "Fri, 30 May 2025 21:30:15 GMT"} })
4182025-05-30T21:30:16.531ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4192025-05-30T21:30:16.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4202025-05-30T21:30:16.534ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4212025-05-30T21:30:16.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4222025-05-30T21:30:16.538ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4232025-05-30T21:30:16.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4242025-05-30T21:30:16.541ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4252025-05-30T21:30:16.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4262025-05-30T21:30:16.548ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 609.104706ms iteration = 9 status = {}
4272025-05-30T21:30:18.774ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 331 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 9bd92e82-3477-4591-90f3-a9c234c9128a response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
4282025-05-30T21:30:25.938ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 26 reason = Timeout
4292025-05-30T21:30:25.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 26 reason = Timeout
4302025-05-30T21:30:25.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4312025-05-30T21:30:25.939ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4322025-05-30T21:30:25.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4332025-05-30T21:30:25.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4342025-05-30T21:30:25.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4352025-05-30T21:30:25.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4362025-05-30T21:30:25.976ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.506411ms iteration = 26 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
4372025-05-30T21:30:25.977ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4382025-05-30T21:30:25.980ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4392025-05-30T21:30:25.980ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4402025-05-30T21:30:25.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4412025-05-30T21:30:25.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4422025-05-30T21:30:25.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4432025-05-30T21:30:25.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4442025-05-30T21:30:25.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
4452025-05-30T21:30:25.987ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4462025-05-30T21:30:25.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4472025-05-30T21:30:25.998ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
4482025-05-30T21:30:25.998ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 60.07546ms iteration = 26 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
4492025-05-30T21:30:28.774ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 350 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 271f5a9f-61d6-436a-9bb0-bc18579e1789 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
4502025-05-30T21:30:35.939ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 27 reason = Timeout
4512025-05-30T21:30:35.939ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4522025-05-30T21:30:35.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 27 reason = Timeout
4532025-05-30T21:30:35.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4542025-05-30T21:30:35.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4552025-05-30T21:30:35.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4562025-05-30T21:30:35.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4572025-05-30T21:30:35.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4582025-05-30T21:30:35.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4592025-05-30T21:30:35.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4602025-05-30T21:30:35.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4612025-05-30T21:30:35.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4622025-05-30T21:30:35.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4632025-05-30T21:30:35.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4642025-05-30T21:30:35.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4652025-05-30T21:30:35.950ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
4662025-05-30T21:30:35.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4672025-05-30T21:30:35.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4682025-05-30T21:30:35.962ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
4692025-05-30T21:30:35.962ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.617839ms iteration = 27 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
4702025-05-30T21:30:35.974ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.521326ms iteration = 27 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
4712025-05-30T21:30:38.774ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 382 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = a31cee23-1242-4954-b958-6e0778b929bd response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
4722025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 28 reason = Timeout
4732025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4742025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 10 reason = Timeout
4752025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 10 reason = Timeout
4762025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
4772025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4782025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 10 reason = Timeout
4792025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
4802025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
4812025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 10 reason = Timeout
4822025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 10 reason = Timeout
4832025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
4842025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4852025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 10 reason = Timeout
4862025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
4872025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 10 reason = Timeout
4882025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
4892025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 10 reason = Timeout
4902025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
4912025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 10 reason = Timeout
4922025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
4932025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 10 reason = Timeout
4942025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
4952025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 10 reason = Timeout
4962025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
4972025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 10 reason = Timeout
4982025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 28 reason = Timeout
4992025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5002025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5012025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 10 reason = Timeout
5022025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 10 reason = Timeout
5032025-05-30T21:30:45.939ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
5042025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
5052025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 11 reason = Timeout
5062025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5072025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5082025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 10 reason = Timeout
5092025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
5102025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 10 reason = Timeout
5112025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5122025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
5132025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
5142025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
5152025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5162025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5172025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
5182025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
5192025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
5202025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
5212025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5222025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5232025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5242025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
5252025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
5262025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5272025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5282025-05-30T21:30:45.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5292025-05-30T21:30:45.944ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 5.572074ms iteration = 10 status = {"errors":[],"requests_created_ok":[]}
5302025-05-30T21:30:45.945ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.653775ms iteration = 10 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
5312025-05-30T21:30:45.945ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5322025-05-30T21:30:45.945ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
5332025-05-30T21:30:45.945ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5342025-05-30T21:30:45.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5352025-05-30T21:30:45.947ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1380, tv_nsec: 886935778 } })
5362025-05-30T21:30:45.948ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
5372025-05-30T21:30:45.949ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5382025-05-30T21:30:45.950ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1380, tv_nsec: 889708070 } })
5392025-05-30T21:30:45.950ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
5402025-05-30T21:30:45.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5412025-05-30T21:30:45.951ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5422025-05-30T21:30:45.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5432025-05-30T21:30:45.954ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5442025-05-30T21:30:45.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5452025-05-30T21:30:45.993ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 54.712429ms iteration = 10 status = {"errors":[],"finish_invoked_ok":[]}
5462025-05-30T21:30:45.997ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
5472025-05-30T21:30:45.997ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5482025-05-30T21:30:45.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
5492025-05-30T21:30:45.998ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5502025-05-30T21:30:45.998ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5512025-05-30T21:30:45.999ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5522025-05-30T21:30:45.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5532025-05-30T21:30:45.999ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 60.423584ms iteration = 10 status = {"errors":[],"garbage_collect_requested":[]}
5542025-05-30T21:30:46.000ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5552025-05-30T21:30:46.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5562025-05-30T21:30:46.001ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5572025-05-30T21:30:46.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5582025-05-30T21:30:46.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
5592025-05-30T21:30:46.002ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5602025-05-30T21:30:46.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5612025-05-30T21:30:46.002ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5622025-05-30T21:30:46.002ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5632025-05-30T21:30:46.002ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
5642025-05-30T21:30:46.002ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5652025-05-30T21:30:46.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
5662025-05-30T21:30:46.003ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5672025-05-30T21:30:46.003ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5682025-05-30T21:30:46.003ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5692025-05-30T21:30:46.003ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5702025-05-30T21:30:46.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5712025-05-30T21:30:46.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5722025-05-30T21:30:46.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5732025-05-30T21:30:46.003ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
5742025-05-30T21:30:46.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
5752025-05-30T21:30:46.004ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5762025-05-30T21:30:46.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5772025-05-30T21:30:46.005ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
5782025-05-30T21:30:46.005ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5792025-05-30T21:30:46.005ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
5802025-05-30T21:30:46.005ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
5812025-05-30T21:30:46.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5822025-05-30T21:30:46.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5832025-05-30T21:30:46.008ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5842025-05-30T21:30:46.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5852025-05-30T21:30:46.008ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1380, tv_nsec: 947962139 } })
5862025-05-30T21:30:46.009ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
5872025-05-30T21:30:46.009ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5882025-05-30T21:30:46.010ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5892025-05-30T21:30:46.012ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
5902025-05-30T21:30:46.012ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5912025-05-30T21:30:46.012ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5922025-05-30T21:30:46.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
5932025-05-30T21:30:46.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5942025-05-30T21:30:46.012ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
5952025-05-30T21:30:46.012ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5962025-05-30T21:30:46.013ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7914b0f1-d4be-49f0-9a76-4b4bc6794928", "content-length": "26", "date": "Fri, 30 May 2025 21:30:45 GMT"} })
5972025-05-30T21:30:46.013ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5982025-05-30T21:30:46.014ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
5992025-05-30T21:30:46.014ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6002025-05-30T21:30:46.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6012025-05-30T21:30:46.015ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6022025-05-30T21:30:46.025ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
6032025-05-30T21:30:46.025ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 86.489173ms iteration = 28 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
6042025-05-30T21:30:46.031ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
6052025-05-30T21:30:46.031ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6062025-05-30T21:30:46.031ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "38bc8341-42e3-474a-aeff-a2295167e64b", "content-length": "26", "date": "Fri, 30 May 2025 21:30:45 GMT"} })
6072025-05-30T21:30:46.031ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6082025-05-30T21:30:46.033ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6092025-05-30T21:30:46.035ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 95.841941ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6102025-05-30T21:30:46.036ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6112025-05-30T21:30:46.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6122025-05-30T21:30:46.037ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
6132025-05-30T21:30:46.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
6142025-05-30T21:30:46.042ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6152025-05-30T21:30:46.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6162025-05-30T21:30:46.043ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 103.843543ms iteration = 10 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
6172025-05-30T21:30:46.053ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6182025-05-30T21:30:46.053ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
6192025-05-30T21:30:46.053ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 114.040419ms iteration = 10 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
6202025-05-30T21:30:46.054ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 115.248103ms iteration = 28 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
6212025-05-30T21:30:46.056ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6222025-05-30T21:30:46.058ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 118.920166ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6232025-05-30T21:30:46.058ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
6242025-05-30T21:30:46.058ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6252025-05-30T21:30:46.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6262025-05-30T21:30:46.061ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
6272025-05-30T21:30:46.061ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 122.094052ms iteration = 10 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
6282025-05-30T21:30:46.062ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6292025-05-30T21:30:46.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6302025-05-30T21:30:46.096ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6312025-05-30T21:30:46.100ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
6322025-05-30T21:30:46.100ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6332025-05-30T21:30:46.100ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6342025-05-30T21:30:46.100ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6352025-05-30T21:30:46.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6362025-05-30T21:30:46.102ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 163.315435ms iteration = 10 status = 0
6372025-05-30T21:30:46.104ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
6382025-05-30T21:30:46.104ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6392025-05-30T21:30:46.105ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0481ccb-658d-4ff4-abe6-b9e125d77d1c", "content-length": "26", "date": "Fri, 30 May 2025 21:30:46 GMT"} })
6402025-05-30T21:30:46.105ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6412025-05-30T21:30:46.107ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
6422025-05-30T21:30:46.115ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 176.274754ms iteration = 10 status = {}
6432025-05-30T21:30:46.124ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6442025-05-30T21:30:46.127ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
6452025-05-30T21:30:46.127ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6462025-05-30T21:30:46.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6472025-05-30T21:30:46.133ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6482025-05-30T21:30:46.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6492025-05-30T21:30:46.136ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 196.698229ms iteration = 10 status = null
6502025-05-30T21:30:46.174ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 234.840797ms iteration = 11 status = {"physical_disks_added":0}
6512025-05-30T21:30:46.175ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
6522025-05-30T21:30:46.175ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 236.223053ms iteration = 10 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
6532025-05-30T21:30:46.243ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6542025-05-30T21:30:46.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6552025-05-30T21:30:46.245ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6562025-05-30T21:30:46.246ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
6572025-05-30T21:30:46.246ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6582025-05-30T21:30:46.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6592025-05-30T21:30:46.253ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
6602025-05-30T21:30:46.253ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
6612025-05-30T21:30:46.253ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca57a45a-e8be-4e36-92db-6152878c90a8", "content-length": "372", "date": "Fri, 30 May 2025 21:30:45 GMT"} })
6622025-05-30T21:30:46.253ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6632025-05-30T21:30:46.254ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6642025-05-30T21:30:46.255ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6652025-05-30T21:30:46.257ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
6662025-05-30T21:30:46.257ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6672025-05-30T21:30:46.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6682025-05-30T21:30:46.260ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6692025-05-30T21:30:46.260ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6702025-05-30T21:30:46.263ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
6712025-05-30T21:30:46.263ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6722025-05-30T21:30:46.263ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6732025-05-30T21:30:46.275ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
6742025-05-30T21:30:46.275ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
6752025-05-30T21:30:46.275ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
6762025-05-30T21:30:46.275ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
6772025-05-30T21:30:46.275ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6782025-05-30T21:30:46.275ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6792025-05-30T21:30:46.275ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6802025-05-30T21:30:46.275ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
6812025-05-30T21:30:46.275ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6822025-05-30T21:30:46.275ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
6832025-05-30T21:30:46.276ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
6842025-05-30T21:30:46.276ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6852025-05-30T21:30:46.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6862025-05-30T21:30:46.278ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
6872025-05-30T21:30:46.278ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
6882025-05-30T21:30:46.308ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
6892025-05-30T21:30:46.308ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
6902025-05-30T21:30:46.350ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
6912025-05-30T21:30:46.384ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "23042c09-74cf-4d46-bbdc-da47d556b6a5", "date": "Fri, 30 May 2025 21:30:45 GMT"} })
6922025-05-30T21:30:46.384ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
6932025-05-30T21:30:46.384ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
6942025-05-30T21:30:46.385ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "aa582fb5-bcde-497c-978d-7f423cde7992", "date": "Fri, 30 May 2025 21:30:45 GMT"} })
6952025-05-30T21:30:46.385ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6962025-05-30T21:30:46.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6972025-05-30T21:30:46.385ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ef997ef-e229-411e-89d8-30d019978d12", "content-length": "682", "date": "Fri, 30 May 2025 21:30:45 GMT"} })
6982025-05-30T21:30:46.385ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "83b78cfe-b51f-4b56-bf7b-0d1c4182e866", "content-length": "691", "date": "Fri, 30 May 2025 21:30:46 GMT"} })
6992025-05-30T21:30:46.385ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7002025-05-30T21:30:46.385ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
7012025-05-30T21:30:46.385ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
7022025-05-30T21:30:46.385ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
7032025-05-30T21:30:46.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7042025-05-30T21:30:46.387ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 448.210008ms iteration = 10 status = {}
7052025-05-30T21:30:46.388ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7062025-05-30T21:30:46.388ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7072025-05-30T21:30:46.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7082025-05-30T21:30:46.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7092025-05-30T21:30:46.392ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7102025-05-30T21:30:46.392ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7112025-05-30T21:30:46.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7122025-05-30T21:30:46.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7132025-05-30T21:30:46.394ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7142025-05-30T21:30:46.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7152025-05-30T21:30:46.396ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
7162025-05-30T21:30:46.396ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
7172025-05-30T21:30:46.396ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "91a81004-fa20-4adb-9534-7d3a56e03062", "date": "Fri, 30 May 2025 21:30:46 GMT"} })
7182025-05-30T21:30:46.398ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 459.136024ms iteration = 10 status = {}
7192025-05-30T21:30:46.401ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 462.081327ms iteration = 10 status = {}
7202025-05-30T21:30:48.775ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 341 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = a4fd34d5-a7d0-4f2f-acc1-c0f52fdfaaf1 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
7212025-05-30T21:30:55.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 29 reason = Timeout
7222025-05-30T21:30:55.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7232025-05-30T21:30:55.939ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
7242025-05-30T21:30:55.939ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7252025-05-30T21:30:55.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7262025-05-30T21:30:55.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7272025-05-30T21:30:55.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7282025-05-30T21:30:55.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7292025-05-30T21:30:55.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7302025-05-30T21:30:55.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7312025-05-30T21:30:55.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7322025-05-30T21:30:55.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7332025-05-30T21:30:55.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7342025-05-30T21:30:55.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
7352025-05-30T21:30:55.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7362025-05-30T21:30:55.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7372025-05-30T21:30:55.962ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
7382025-05-30T21:30:55.962ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.893732ms iteration = 29 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
7392025-05-30T21:30:55.972ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7402025-05-30T21:30:55.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7412025-05-30T21:30:56.010ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 71.201658ms iteration = 29 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7422025-05-30T21:30:58.774ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 383 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 8b36f0ba-cdec-4ab7-9a3a-c7a6cae1ce21 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
7432025-05-30T21:31:05.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 30 reason = Timeout
7442025-05-30T21:31:05.939ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
7452025-05-30T21:31:05.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7462025-05-30T21:31:05.939ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7472025-05-30T21:31:05.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7482025-05-30T21:31:05.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7492025-05-30T21:31:05.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7502025-05-30T21:31:05.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7512025-05-30T21:31:05.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7522025-05-30T21:31:05.943ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.714055ms iteration = 30 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7532025-05-30T21:31:05.943ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7542025-05-30T21:31:05.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7552025-05-30T21:31:05.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7562025-05-30T21:31:05.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7572025-05-30T21:31:05.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7582025-05-30T21:31:05.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7592025-05-30T21:31:05.950ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
7602025-05-30T21:31:05.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7612025-05-30T21:31:05.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7622025-05-30T21:31:05.959ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
7632025-05-30T21:31:05.959ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.91487ms iteration = 30 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
7642025-05-30T21:31:08.775ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 347 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 238a5d96-7018-4181-b103-dd97dcee7039 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
7652025-05-30T21:31:15.938ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
7662025-05-30T21:31:15.938ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
7672025-05-30T21:31:15.938ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7682025-05-30T21:31:15.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7692025-05-30T21:31:15.938ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
7702025-05-30T21:31:15.938ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7712025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 11 reason = Timeout
7722025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
7732025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 6 reason = Timeout
7742025-05-30T21:31:15.939ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-30 21:21:15.939020635 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
7752025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
7762025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
7772025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7782025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 8 reason = Timeout
7792025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 6 reason = Timeout
7802025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
7812025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
7822025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
7832025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 6 reason = Timeout
7842025-05-30T21:31:15.939ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
7852025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 11 reason = Timeout
7862025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
7872025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 6 reason = Timeout
7882025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
7892025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7902025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 11 reason = Timeout
7912025-05-30T21:31:15.939ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
7922025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
7932025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7942025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7952025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 11 reason = Timeout
7962025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7972025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7982025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
7992025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 11 reason = Timeout
8002025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8012025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
8022025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 12 reason = Timeout
8032025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8042025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 8 reason = Timeout
8052025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 6 reason = Timeout
8062025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
8072025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 31 reason = Timeout
8082025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8092025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
8102025-05-30T21:31:15.939ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e
8112025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.04\u{b5}s iteration = 7 status = {"enabled":false,"target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e"}
8122025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 2 reason = Timeout
8132025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 2 reason = Timeout
8142025-05-30T21:31:15.939ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
8152025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
8162025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 11 reason = Timeout
8172025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8182025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
8192025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 3 reason = Timeout
8202025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
8212025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 11 reason = Timeout
8222025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
8232025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
8242025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 6 reason = Timeout
8252025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
8262025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
8272025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 11 reason = Timeout
8282025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 6 reason = Timeout
8292025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
8302025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
8312025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
8322025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
8332025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
8342025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8352025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
8362025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 11 reason = Timeout
8372025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
8382025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 11 reason = Timeout
8392025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
8402025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
8412025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
8422025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8432025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 11 reason = Timeout
8442025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
8452025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
8462025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8472025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
8482025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8492025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8502025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8512025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
8522025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
8532025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8542025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
8552025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8562025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
8572025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
8582025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8592025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
8602025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8612025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
8622025-05-30T21:31:15.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
8632025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8642025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
8652025-05-30T21:31:15.940ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0413f6b4-86a1-42a1-919a-e8f42ff9f638.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("acb1452d-0a02-4b36-a462-c8ce2f96a488.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:2::1))) }], valid_until: Instant { tv_sec: 1410, tv_nsec: 878719315 } })
8662025-05-30T21:31:15.940ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2a161087-f464-41f8-b020-0f9e83fd5351.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1410, tv_nsec: 878719385 } })
8672025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
8682025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8692025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
8702025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
8712025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8722025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
8732025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8742025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
8752025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
8762025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8772025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
8782025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 971.251\u{b5}s iteration = 6 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
8792025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } dns_group = internal
8802025-05-30T21:31:15.940ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.155623ms iteration = 6 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353"]}
8812025-05-30T21:31:15.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8822025-05-30T21:31:15.943ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 4.488302ms iteration = 11 status = {"errors":[],"requests_created_ok":[]}
8832025-05-30T21:31:15.944ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8842025-05-30T21:31:15.944ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.061338ms iteration = 11 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
8852025-05-30T21:31:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8862025-05-30T21:31:15.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8872025-05-30T21:31:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8882025-05-30T21:31:15.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
8892025-05-30T21:31:15.946ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8902025-05-30T21:31:15.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8912025-05-30T21:31:15.946ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8922025-05-30T21:31:15.947ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8932025-05-30T21:31:15.947ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-30 21:21:15.939020635 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
8942025-05-30T21:31:15.947ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 8.139764ms iteration = 6 status = {"expiration":"2025-05-30T21:21:15.939020635Z","pruned":[]}
8952025-05-30T21:31:15.947ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8962025-05-30T21:31:15.950ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
8972025-05-30T21:31:15.950ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8982025-05-30T21:31:15.950ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
8992025-05-30T21:31:15.950ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9002025-05-30T21:31:15.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9012025-05-30T21:31:15.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9022025-05-30T21:31:15.950ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9032025-05-30T21:31:15.950ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9042025-05-30T21:31:15.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9052025-05-30T21:31:15.952ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9062025-05-30T21:31:15.952ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1410, tv_nsec: 891377701 } })
9072025-05-30T21:31:15.952ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
9082025-05-30T21:31:15.953ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9092025-05-30T21:31:15.953ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1410, tv_nsec: 892260721 } })
9102025-05-30T21:31:15.953ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
9112025-05-30T21:31:15.954ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9122025-05-30T21:31:15.954ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9132025-05-30T21:31:15.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9142025-05-30T21:31:15.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
9152025-05-30T21:31:15.955ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
9162025-05-30T21:31:15.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
9172025-05-30T21:31:15.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9182025-05-30T21:31:15.963ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 23.968775ms iteration = 11 status = {"errors":[],"garbage_collect_requested":[]}
9192025-05-30T21:31:15.964ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9202025-05-30T21:31:15.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9212025-05-30T21:31:15.965ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
9222025-05-30T21:31:15.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9232025-05-30T21:31:15.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9242025-05-30T21:31:15.967ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:202 num_receivers = 0
9252025-05-30T21:31:15.967ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 28.006652ms iteration = 6 status = {"by_rx":{},"error":null}
9262025-05-30T21:31:15.967ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
9272025-05-30T21:31:15.967ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9282025-05-30T21:31:15.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9292025-05-30T21:31:15.968ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 29.248526ms iteration = 6 status = {"errors":[],"found_port_ok":[]}
9302025-05-30T21:31:15.970ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9312025-05-30T21:31:15.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9322025-05-30T21:31:15.970ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9332025-05-30T21:31:15.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9342025-05-30T21:31:15.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9352025-05-30T21:31:15.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9362025-05-30T21:31:15.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9372025-05-30T21:31:15.977ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1410, tv_nsec: 916327467 } })
9382025-05-30T21:31:15.977ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
9392025-05-30T21:31:15.977ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9402025-05-30T21:31:15.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9412025-05-30T21:31:15.982ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
9422025-05-30T21:31:15.982ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
9432025-05-30T21:31:15.982ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
9442025-05-30T21:31:15.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9452025-05-30T21:31:15.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9462025-05-30T21:31:15.982ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9472025-05-30T21:31:15.982ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9482025-05-30T21:31:15.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9492025-05-30T21:31:15.982ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9502025-05-30T21:31:15.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9512025-05-30T21:31:15.984ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9522025-05-30T21:31:15.984ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
9532025-05-30T21:31:15.984ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9542025-05-30T21:31:15.984ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9552025-05-30T21:31:15.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
9562025-05-30T21:31:15.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9572025-05-30T21:31:15.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9582025-05-30T21:31:15.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9592025-05-30T21:31:15.986ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9602025-05-30T21:31:15.987ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9612025-05-30T21:31:15.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9622025-05-30T21:31:15.991ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9632025-05-30T21:31:15.994ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9642025-05-30T21:31:15.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9652025-05-30T21:31:15.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9662025-05-30T21:31:15.999ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9672025-05-30T21:31:16.000ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9682025-05-30T21:31:16.001ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9692025-05-30T21:31:16.004ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
9702025-05-30T21:31:16.004ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9712025-05-30T21:31:16.004ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
9722025-05-30T21:31:16.004ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9732025-05-30T21:31:16.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9742025-05-30T21:31:16.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9752025-05-30T21:31:16.005ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9762025-05-30T21:31:16.005ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9772025-05-30T21:31:16.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9782025-05-30T21:31:16.007ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 67.858799ms iteration = 11 status = {"errors":[],"finish_invoked_ok":[]}
9792025-05-30T21:31:16.008ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9802025-05-30T21:31:16.008ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 69.436788ms iteration = 6 status = {"dispatched":[],"errors":[],"glob_version":"144.0.0","globs_reprocessed":{},"no_receivers":[]}
9812025-05-30T21:31:16.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9822025-05-30T21:31:16.010ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
9832025-05-30T21:31:16.011ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 71.861985ms iteration = 11 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
9842025-05-30T21:31:16.015ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
9852025-05-30T21:31:16.015ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 76.56759ms iteration = 31 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
9862025-05-30T21:31:16.016ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9872025-05-30T21:31:16.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9882025-05-30T21:31:16.017ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9892025-05-30T21:31:16.017ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9902025-05-30T21:31:16.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9912025-05-30T21:31:16.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9922025-05-30T21:31:16.017ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
9932025-05-30T21:31:16.017ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
9942025-05-30T21:31:16.017ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 78.647754ms iteration = 6 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
9952025-05-30T21:31:16.018ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9962025-05-30T21:31:16.018ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 79.487233ms iteration = 31 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
9972025-05-30T21:31:16.019ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9982025-05-30T21:31:16.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9992025-05-30T21:31:16.020ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
10002025-05-30T21:31:16.020ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10012025-05-30T21:31:16.020ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
10022025-05-30T21:31:16.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10032025-05-30T21:31:16.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
10042025-05-30T21:31:16.021ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10052025-05-30T21:31:16.022ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10062025-05-30T21:31:16.022ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10072025-05-30T21:31:16.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10082025-05-30T21:31:16.023ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
10092025-05-30T21:31:16.023ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
10102025-05-30T21:31:16.023ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
10112025-05-30T21:31:16.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
10122025-05-30T21:31:16.024ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10132025-05-30T21:31:16.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10142025-05-30T21:31:16.024ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
10152025-05-30T21:31:16.024ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10162025-05-30T21:31:16.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10172025-05-30T21:31:16.025ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
10182025-05-30T21:31:16.025ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10192025-05-30T21:31:16.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10202025-05-30T21:31:16.026ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 87.364633ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10212025-05-30T21:31:16.027ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10222025-05-30T21:31:16.028ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10232025-05-30T21:31:16.028ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
10242025-05-30T21:31:16.028ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
10252025-05-30T21:31:16.028ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
10262025-05-30T21:31:16.029ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10272025-05-30T21:31:16.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
10282025-05-30T21:31:16.030ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10292025-05-30T21:31:16.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10302025-05-30T21:31:16.030ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10312025-05-30T21:31:16.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10322025-05-30T21:31:16.031ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
10332025-05-30T21:31:16.031ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10342025-05-30T21:31:16.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10352025-05-30T21:31:16.031ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
10362025-05-30T21:31:16.031ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10372025-05-30T21:31:16.031ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
10382025-05-30T21:31:16.031ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10392025-05-30T21:31:16.031ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
10402025-05-30T21:31:16.031ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
10412025-05-30T21:31:16.031ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 92.814276ms iteration = 6 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
10422025-05-30T21:31:16.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10432025-05-30T21:31:16.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10442025-05-30T21:31:16.035ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10452025-05-30T21:31:16.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10462025-05-30T21:31:16.038ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10472025-05-30T21:31:16.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10482025-05-30T21:31:16.039ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10492025-05-30T21:31:16.040ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10502025-05-30T21:31:16.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10512025-05-30T21:31:16.042ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
10522025-05-30T21:31:16.042ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10532025-05-30T21:31:16.042ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10542025-05-30T21:31:16.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10552025-05-30T21:31:16.044ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
10562025-05-30T21:31:16.044ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
10572025-05-30T21:31:16.044ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
10582025-05-30T21:31:16.044ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10592025-05-30T21:31:16.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10602025-05-30T21:31:16.044ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 105.711484ms iteration = 11 status = 0
10612025-05-30T21:31:16.048ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
10622025-05-30T21:31:16.048ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10632025-05-30T21:31:16.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10642025-05-30T21:31:16.052ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10652025-05-30T21:31:16.055ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
10662025-05-30T21:31:16.055ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10672025-05-30T21:31:16.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10682025-05-30T21:31:16.056ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10692025-05-30T21:31:16.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10702025-05-30T21:31:16.057ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
10712025-05-30T21:31:16.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
10722025-05-30T21:31:16.060ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
10732025-05-30T21:31:16.060ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 0 version = 2
10742025-05-30T21:31:16.060ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10752025-05-30T21:31:16.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10762025-05-30T21:31:16.060ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10772025-05-30T21:31:16.061ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10782025-05-30T21:31:16.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10792025-05-30T21:31:16.062ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10802025-05-30T21:31:16.062ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 122.968503ms iteration = 11 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
10812025-05-30T21:31:16.062ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10822025-05-30T21:31:16.063ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
10832025-05-30T21:31:16.063ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
10842025-05-30T21:31:16.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
10852025-05-30T21:31:16.065ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10862025-05-30T21:31:16.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10872025-05-30T21:31:16.068ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10882025-05-30T21:31:16.071ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
10892025-05-30T21:31:16.071ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
10902025-05-30T21:31:16.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
10912025-05-30T21:31:16.071ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
10922025-05-30T21:31:16.071ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10932025-05-30T21:31:16.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10942025-05-30T21:31:16.074ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10952025-05-30T21:31:16.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10962025-05-30T21:31:16.075ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
10972025-05-30T21:31:16.075ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
10982025-05-30T21:31:16.075ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10992025-05-30T21:31:16.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11002025-05-30T21:31:16.076ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 1 version = 2
11012025-05-30T21:31:16.076ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
11022025-05-30T21:31:16.076ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-30 21:26:18.957109 UTC dns_group = external generation = 2
11032025-05-30T21:31:16.076ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 136.845772ms iteration = 6 status = {"generation":2}
11042025-05-30T21:31:16.076ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11052025-05-30T21:31:16.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11062025-05-30T21:31:16.078ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11072025-05-30T21:31:16.078ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11082025-05-30T21:31:16.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11092025-05-30T21:31:16.081ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
11102025-05-30T21:31:16.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
11112025-05-30T21:31:16.083ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
11122025-05-30T21:31:16.083ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11132025-05-30T21:31:16.083ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11142025-05-30T21:31:16.095ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 155.674117ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
11152025-05-30T21:31:16.095ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11162025-05-30T21:31:16.097ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11172025-05-30T21:31:16.098ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
11182025-05-30T21:31:16.098ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
11192025-05-30T21:31:16.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
11202025-05-30T21:31:16.099ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
11212025-05-30T21:31:16.099ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
11222025-05-30T21:31:16.099ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-30 21:26:17.226115 UTC dns_group = internal generation = 1
11232025-05-30T21:31:16.099ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 160.591184ms iteration = 6 status = {"generation":1}
11242025-05-30T21:31:16.101ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11252025-05-30T21:31:16.102ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
11262025-05-30T21:31:16.102ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11272025-05-30T21:31:16.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11282025-05-30T21:31:16.104ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
11292025-05-30T21:31:16.104ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
11302025-05-30T21:31:16.104ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
11312025-05-30T21:31:16.105ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
11322025-05-30T21:31:16.105ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
11332025-05-30T21:31:16.105ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
11342025-05-30T21:31:16.105ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 165.636241ms iteration = 11 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
11352025-05-30T21:31:16.105ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11362025-05-30T21:31:16.107ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-30T21:26:13.986396Z, time_modified: 2025-05-30T21:26:13.986396Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }) }
11372025-05-30T21:31:16.107ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 167.762037ms iteration = 6 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]}},"default_endpoint":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
11382025-05-30T21:31:16.111ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11392025-05-30T21:31:16.111ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
11402025-05-30T21:31:16.111ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11412025-05-30T21:31:16.111ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11422025-05-30T21:31:16.111ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9490ee0d-1740-44d7-b716-3f2eb54c9821", "content-length": "26", "date": "Fri, 30 May 2025 21:31:16 GMT"} })
11432025-05-30T21:31:16.111ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11442025-05-30T21:31:16.114ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11452025-05-30T21:31:16.118ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11462025-05-30T21:31:16.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11472025-05-30T21:31:16.121ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11482025-05-30T21:31:16.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11492025-05-30T21:31:16.123ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
11502025-05-30T21:31:16.127ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11512025-05-30T21:31:16.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11522025-05-30T21:31:16.128ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 189.365144ms iteration = 3 status = {"blueprint_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","inventory_collection_id":"0165edca-c075-4a18-9f51-5e948173b5ec","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
11532025-05-30T21:31:16.131ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11542025-05-30T21:31:16.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11552025-05-30T21:31:16.137ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
11562025-05-30T21:31:16.138ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
11572025-05-30T21:31:16.167ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11582025-05-30T21:31:16.167ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11592025-05-30T21:31:16.168ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
11602025-05-30T21:31:16.168ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11612025-05-30T21:31:16.168ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3752a4fa-225d-4652-9123-44cce2a26b43", "content-length": "26", "date": "Fri, 30 May 2025 21:31:16 GMT"} })
11622025-05-30T21:31:16.169ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11632025-05-30T21:31:16.169ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
11642025-05-30T21:31:16.171ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11652025-05-30T21:31:16.172ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 232.881744ms iteration = 11 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
11662025-05-30T21:31:16.194ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11672025-05-30T21:31:16.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11682025-05-30T21:31:16.195ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
11692025-05-30T21:31:16.195ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11702025-05-30T21:31:16.196ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dc51b82-5b1b-452a-b0de-ad405738fa52", "content-length": "26", "date": "Fri, 30 May 2025 21:31:16 GMT"} })
11712025-05-30T21:31:16.196ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11722025-05-30T21:31:16.198ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
11732025-05-30T21:31:16.245ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 306.378768ms iteration = 12 status = {"physical_disks_added":0}
11742025-05-30T21:31:16.270ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
11752025-05-30T21:31:16.333ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
11762025-05-30T21:31:16.333ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 394.257528ms iteration = 6 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
11772025-05-30T21:31:16.334ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11782025-05-30T21:31:16.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11792025-05-30T21:31:16.336ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11802025-05-30T21:31:16.337ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11812025-05-30T21:31:16.361ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
11822025-05-30T21:31:16.361ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
11832025-05-30T21:31:16.361ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "93d7d9f7-5ed1-4e06-91fa-8b5f9e470c94", "content-length": "372", "date": "Fri, 30 May 2025 21:31:15 GMT"} })
11842025-05-30T21:31:16.361ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11852025-05-30T21:31:16.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11862025-05-30T21:31:16.362ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11872025-05-30T21:31:16.363ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11882025-05-30T21:31:16.364ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 424.94241ms iteration = 11 status = null
11892025-05-30T21:31:16.365ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11902025-05-30T21:31:16.365ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11912025-05-30T21:31:16.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11922025-05-30T21:31:16.367ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11932025-05-30T21:31:16.369ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
11942025-05-30T21:31:16.369ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11952025-05-30T21:31:16.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11962025-05-30T21:31:16.369ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11972025-05-30T21:31:16.369ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11982025-05-30T21:31:16.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11992025-05-30T21:31:16.398ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 459.387155ms iteration = 11 status = {}
12002025-05-30T21:31:16.405ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12012025-05-30T21:31:16.405ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12022025-05-30T21:31:16.411ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12032025-05-30T21:31:16.411ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12042025-05-30T21:31:16.412ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
12052025-05-30T21:31:16.413ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
12062025-05-30T21:31:16.413ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "d0480467-9320-4108-b868-c132214e7387", "date": "Fri, 30 May 2025 21:31:15 GMT"} })
12072025-05-30T21:31:16.413ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12082025-05-30T21:31:16.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12092025-05-30T21:31:16.439ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12102025-05-30T21:31:16.441ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
12112025-05-30T21:31:16.441ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12122025-05-30T21:31:16.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12132025-05-30T21:31:16.443ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12142025-05-30T21:31:16.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12152025-05-30T21:31:16.459ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
12162025-05-30T21:31:16.460ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "37ed507b-1314-4a0d-899e-12adb82dafa3", "date": "Fri, 30 May 2025 21:31:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12172025-05-30T21:31:16.460ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12182025-05-30T21:31:16.462ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "5e1ce3bc-5c58-4bd0-a3d2-bc8cdb312257", "date": "Fri, 30 May 2025 21:31:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12192025-05-30T21:31:16.462ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12202025-05-30T21:31:16.463ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12212025-05-30T21:31:16.463ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 524.607105ms iteration = 8 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
12222025-05-30T21:31:16.499ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
12232025-05-30T21:31:16.500ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
12242025-05-30T21:31:16.500ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "b746f5d2-394b-4007-a418-6647bf3f111f", "content-length": "691", "date": "Fri, 30 May 2025 21:31:16 GMT"} })
12252025-05-30T21:31:16.500ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
12262025-05-30T21:31:16.500ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
12272025-05-30T21:31:16.500ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
12282025-05-30T21:31:16.502ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12292025-05-30T21:31:16.502ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 563.194508ms iteration = 11 status = {}
12302025-05-30T21:31:16.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12312025-05-30T21:31:16.502ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12322025-05-30T21:31:16.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12332025-05-30T21:31:16.506ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12342025-05-30T21:31:16.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12352025-05-30T21:31:16.509ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
12362025-05-30T21:31:16.509ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
12372025-05-30T21:31:16.509ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "d80a826c-bc02-4181-9c72-9f61c16a10aa", "date": "Fri, 30 May 2025 21:31:16 GMT"} })
12382025-05-30T21:31:16.511ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 572.553326ms iteration = 11 status = {}
12392025-05-30T21:31:16.536ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12402025-05-30T21:31:16.540ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
12412025-05-30T21:31:16.540ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12422025-05-30T21:31:16.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12432025-05-30T21:31:16.547ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12442025-05-30T21:31:16.549ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
12452025-05-30T21:31:16.549ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12462025-05-30T21:31:16.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12472025-05-30T21:31:16.553ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/artifacts-config
12482025-05-30T21:31:16.554ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12492025-05-30T21:31:16.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12502025-05-30T21:31:16.555ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts-config", status: 204, headers: {"x-request-id": "b8365164-cf27-4e57-9677-d349edb818a0", "date": "Fri, 30 May 2025 21:31:16 GMT"} })
12512025-05-30T21:31:16.556ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12522025-05-30T21:31:16.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12532025-05-30T21:31:16.558ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
12542025-05-30T21:31:16.558ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "ade3e0aa-6be8-4d57-81be-cfa7015b8f74", "content-length": "26", "date": "Fri, 30 May 2025 21:31:15 GMT"} })
12552025-05-30T21:31:16.558ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:673 sled = http://[fd00:1122:3344:101::1]:12345
12562025-05-30T21:31:16.559ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12572025-05-30T21:31:16.559ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12582025-05-30T21:31:16.561ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12592025-05-30T21:31:16.561ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12602025-05-30T21:31:16.562ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 622.764392ms iteration = 2 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"5d926227-6a52-4938-b74d-21ba39bce2f9","time":"2025-05-30T21:31:16.555348283Z"},{"operation":"list","target_sled":"5d926227-6a52-4938-b74d-21ba39bce2f9","time":"2025-05-30T21:31:16.558295177Z"}]}
12612025-05-30T21:31:16.563ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
12622025-05-30T21:31:16.563ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12632025-05-30T21:31:16.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12642025-05-30T21:31:16.563ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "25726591-a9de-4372-892e-ab55d7deab36", "date": "Fri, 30 May 2025 21:31:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
12652025-05-30T21:31:16.564ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
12662025-05-30T21:31:16.564ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12672025-05-30T21:31:16.565ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12682025-05-30T21:31:16.565ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "92537e36-b377-433e-935a-bff7218fd987", "date": "Fri, 30 May 2025 21:31:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
12692025-05-30T21:31:16.565ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
12702025-05-30T21:31:16.566ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12712025-05-30T21:31:16.566ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12722025-05-30T21:31:16.567ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "01808e28-af24-49ea-a1f4-125c2b155cd2", "date": "Fri, 30 May 2025 21:31:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
12732025-05-30T21:31:16.567ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
12742025-05-30T21:31:16.569ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
12752025-05-30T21:31:16.569ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 630.630533ms iteration = 8 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
12762025-05-30T21:31:16.576ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12772025-05-30T21:31:16.576ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12782025-05-30T21:31:16.578ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
12792025-05-30T21:31:16.578ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12802025-05-30T21:31:16.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12812025-05-30T21:31:16.591ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
12822025-05-30T21:31:16.591ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
12832025-05-30T21:31:16.591ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
12842025-05-30T21:31:16.591ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
12852025-05-30T21:31:16.591ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
12862025-05-30T21:31:16.591ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
12872025-05-30T21:31:16.591ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
12882025-05-30T21:31:16.591ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
12892025-05-30T21:31:16.591ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
12902025-05-30T21:31:16.591ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
12912025-05-30T21:31:16.592ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
12922025-05-30T21:31:16.592ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12932025-05-30T21:31:16.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12942025-05-30T21:31:16.594ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
12952025-05-30T21:31:16.594ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
12962025-05-30T21:31:16.606ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12972025-05-30T21:31:16.609ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
12982025-05-30T21:31:16.609ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12992025-05-30T21:31:16.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13002025-05-30T21:31:16.611ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13012025-05-30T21:31:16.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13022025-05-30T21:31:16.612ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13032025-05-30T21:31:16.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13042025-05-30T21:31:16.654ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13052025-05-30T21:31:16.658ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
13062025-05-30T21:31:16.658ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13072025-05-30T21:31:16.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13082025-05-30T21:31:16.665ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 957a8a3c-41fc-4640-b220-a063b1626528, kind: Service { id: 8afbed31-e520-4cc6-aa5c-72f8b96b171a }, name: Name("nexus-8afbed31-e520-4cc6-aa5c-72f8b96b171a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 83])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b2cee132-ec62-413c-b43c-0e9042d338c2, kind: Service { id: bb2ce2e9-85b4-4cff-aedf-120e3ffadec2 }, name: Name("nexus-bb2ce2e9-85b4-4cff-aedf-120e3ffadec2"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 208, 170])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bea3c463-5b2a-46b8-ad8c-4e1a767d4ece, kind: Service { id: ace9a692-9e36-4a57-88cc-937ba7a7ab5e }, name: Name("nexus-ace9a692-9e36-4a57-88cc-937ba7a7ab5e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 2990b628-13b9-47b6-8f08-4b84713240a8, kind: Service { id: 2a161087-f464-41f8-b020-0f9e83fd5351 }, name: Name("external-dns-2a161087-f464-41f8-b020-0f9e83fd5351"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 127])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 535411d1-cf1c-44db-becb-5ce60a31b05b, kind: Service { id: 5c275d18-0bbd-414f-a1ca-4f5fb261fa3a }, name: Name("external-dns-5c275d18-0bbd-414f-a1ca-4f5fb261fa3a"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 225, 1])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
13092025-05-30T21:31:16.665ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13102025-05-30T21:31:16.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13112025-05-30T21:31:16.667ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13122025-05-30T21:31:16.669ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
13132025-05-30T21:31:16.669ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13142025-05-30T21:31:16.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13152025-05-30T21:31:16.671ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
13162025-05-30T21:31:16.671ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 2990b628-13b9-47b6-8f08-4b84713240a8, kind: Service { id: 2a161087-f464-41f8-b020-0f9e83fd5351 }, name: Name("external-dns-2a161087-f464-41f8-b020-0f9e83fd5351"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 127])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 535411d1-cf1c-44db-becb-5ce60a31b05b, kind: Service { id: 5c275d18-0bbd-414f-a1ca-4f5fb261fa3a }, name: Name("external-dns-5c275d18-0bbd-414f-a1ca-4f5fb261fa3a"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 225, 1])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 957a8a3c-41fc-4640-b220-a063b1626528, kind: Service { id: 8afbed31-e520-4cc6-aa5c-72f8b96b171a }, name: Name("nexus-8afbed31-e520-4cc6-aa5c-72f8b96b171a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 83])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b2cee132-ec62-413c-b43c-0e9042d338c2, kind: Service { id: bb2ce2e9-85b4-4cff-aedf-120e3ffadec2 }, name: Name("nexus-bb2ce2e9-85b4-4cff-aedf-120e3ffadec2"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 208, 170])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bea3c463-5b2a-46b8-ad8c-4e1a767d4ece, kind: Service { id: ace9a692-9e36-4a57-88cc-937ba7a7ab5e }, name: Name("nexus-ace9a692-9e36-4a57-88cc-937ba7a7ab5e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
13172025-05-30T21:31:16.671ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
13182025-05-30T21:31:16.710ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "59d2652f-ab4f-47e4-9be8-68802ed58055", "date": "Fri, 30 May 2025 21:31:15 GMT"} })
13192025-05-30T21:31:16.710ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
13202025-05-30T21:31:16.710ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
13212025-05-30T21:31:16.711ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "bccab370-a23e-4ed6-96fb-33b7feda44b6", "content-length": "682", "date": "Fri, 30 May 2025 21:31:16 GMT"} })
13222025-05-30T21:31:16.711ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13232025-05-30T21:31:16.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13242025-05-30T21:31:16.715ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13252025-05-30T21:31:16.715ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13262025-05-30T21:31:16.719ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13272025-05-30T21:31:16.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13282025-05-30T21:31:16.721ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13292025-05-30T21:31:16.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13302025-05-30T21:31:16.728ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 789.442356ms iteration = 11 status = {}
13312025-05-30T21:31:16.729ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 5d926227-6a52-4938-b74d-21ba39bce2f9, time_created: 2025-05-30T21:26:16.127586Z, time_modified: 2025-05-30T21:26:16.127586Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(56094621696)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
13322025-05-30T21:31:16.729ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
13332025-05-30T21:31:16.766ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13342025-05-30T21:31:16.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13352025-05-30T21:31:16.769ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13362025-05-30T21:31:16.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13372025-05-30T21:31:16.771ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13382025-05-30T21:31:16.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
13392025-05-30T21:31:16.776ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
13402025-05-30T21:31:16.777ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "00478234-9c75-4678-8633-426956c6d7ee", "date": "Fri, 30 May 2025 21:31:16 GMT"} })
13412025-05-30T21:31:16.779ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:495
13422025-05-30T21:31:16.779ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 840.142899ms
13432025-05-30T21:31:16.779ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 840.151019ms iteration = 2 status = {}
13442025-05-30T21:31:18.774ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 306 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 1e17a472-d06b-4de1-921f-44a59760601b response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
13452025-05-30T21:31:25.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 32 reason = Timeout
13462025-05-30T21:31:25.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13472025-05-30T21:31:25.939ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
13482025-05-30T21:31:25.939ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13492025-05-30T21:31:25.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13502025-05-30T21:31:25.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13512025-05-30T21:31:25.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13522025-05-30T21:31:25.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
13532025-05-30T21:31:25.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13542025-05-30T21:31:25.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13552025-05-30T21:31:25.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13562025-05-30T21:31:25.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13572025-05-30T21:31:25.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13582025-05-30T21:31:25.949ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
13592025-05-30T21:31:25.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13602025-05-30T21:31:25.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13612025-05-30T21:31:25.960ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
13622025-05-30T21:31:25.960ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.837331ms iteration = 32 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
13632025-05-30T21:31:25.971ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13642025-05-30T21:31:25.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13652025-05-30T21:31:26.003ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.145487ms iteration = 32 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
13662025-05-30T21:31:28.765ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:35849
13672025-05-30T21:31:28.765ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35849 req_id = f1772496-55a3-4bb8-abe9-d435ff65ed3f roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
13682025-05-30T21:31:28.765ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35849 req_id = f1772496-55a3-4bb8-abe9-d435ff65ed3f resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
13692025-05-30T21:31:28.768ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3217 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35849 req_id = f1772496-55a3-4bb8-abe9-d435ff65ed3f response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
13702025-05-30T21:31:28.768ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35849 req_id = bd89815c-5b4e-4ad8-9312-f4385288aadd roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
13712025-05-30T21:31:28.769ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35849 req_id = bd89815c-5b4e-4ad8-9312-f4385288aadd resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
13722025-05-30T21:31:28.770ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 1907 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35849 req_id = bd89815c-5b4e-4ad8-9312-f4385288aadd response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
13732025-05-30T21:31:28.774ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 309 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 915d3901-8a05-472e-b670-61d07eff5919 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
13742025-05-30T21:31:35.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 33 reason = Timeout
13752025-05-30T21:31:35.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13762025-05-30T21:31:35.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
13772025-05-30T21:31:35.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13782025-05-30T21:31:35.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13792025-05-30T21:31:35.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13802025-05-30T21:31:35.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13812025-05-30T21:31:35.945ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
13822025-05-30T21:31:35.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13832025-05-30T21:31:35.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13842025-05-30T21:31:35.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13852025-05-30T21:31:35.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13862025-05-30T21:31:35.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13872025-05-30T21:31:35.952ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
13882025-05-30T21:31:35.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13892025-05-30T21:31:35.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13902025-05-30T21:31:35.963ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
13912025-05-30T21:31:35.963ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.014425ms iteration = 33 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
13922025-05-30T21:31:35.973ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13932025-05-30T21:31:35.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13942025-05-30T21:31:36.003ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 63.674881ms iteration = 33 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
13952025-05-30T21:31:38.775ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 328 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = c90b602e-125a-43ae-b059-a30bb0dd85b6 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
13962025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 34 reason = Timeout
13972025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 12 reason = Timeout
13982025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13992025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 12 reason = Timeout
14002025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
14012025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
14022025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
14032025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
14042025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
14052025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 12 reason = Timeout
14062025-05-30T21:31:45.939ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
14072025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 12 reason = Timeout
14082025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 12 reason = Timeout
14092025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14102025-05-30T21:31:45.939ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14112025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 12 reason = Timeout
14122025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
14132025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
14142025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
14152025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
14162025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
14172025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 12 reason = Timeout
14182025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14192025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 12 reason = Timeout
14202025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
14212025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14222025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 12 reason = Timeout
14232025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
14242025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
14252025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14262025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 12 reason = Timeout
14272025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
14282025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 13 reason = Timeout
14292025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
14302025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
14312025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
14322025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 12 reason = Timeout
14332025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
14342025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
14352025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
14362025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14372025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
14382025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
14392025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14402025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14412025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
14422025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
14432025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
14442025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
14452025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
14462025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14472025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
14482025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
14492025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14502025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
14512025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14522025-05-30T21:31:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
14532025-05-30T21:31:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14542025-05-30T21:31:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14552025-05-30T21:31:45.942ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1440, tv_nsec: 881287287 } })
14562025-05-30T21:31:45.943ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
14572025-05-30T21:31:45.943ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.693482ms iteration = 12 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
14582025-05-30T21:31:45.947ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14592025-05-30T21:31:45.948ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1440, tv_nsec: 887336926 } })
14602025-05-30T21:31:45.949ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
14612025-05-30T21:31:45.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
14622025-05-30T21:31:45.951ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14632025-05-30T21:31:45.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14642025-05-30T21:31:45.955ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14652025-05-30T21:31:45.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14662025-05-30T21:31:45.987ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14672025-05-30T21:31:45.989ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
14682025-05-30T21:31:45.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
14692025-05-30T21:31:45.989ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
14702025-05-30T21:31:45.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
14712025-05-30T21:31:45.991ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14722025-05-30T21:31:45.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
14732025-05-30T21:31:45.991ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
14742025-05-30T21:31:45.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
14752025-05-30T21:31:45.993ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
14762025-05-30T21:31:45.993ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14772025-05-30T21:31:45.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14782025-05-30T21:31:45.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14792025-05-30T21:31:45.995ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14802025-05-30T21:31:45.995ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 55.71584ms iteration = 12 status = {"errors":[],"finish_invoked_ok":[]}
14812025-05-30T21:31:45.996ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
14822025-05-30T21:31:45.996ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
14832025-05-30T21:31:45.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
14842025-05-30T21:31:45.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
14852025-05-30T21:31:45.996ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14862025-05-30T21:31:45.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14872025-05-30T21:31:45.997ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 57.891085ms iteration = 34 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
14882025-05-30T21:31:45.998ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 58.250919ms iteration = 12 status = {"errors":[],"garbage_collect_requested":[]}
14892025-05-30T21:31:45.998ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14902025-05-30T21:31:45.998ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14912025-05-30T21:31:45.998ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
14922025-05-30T21:31:45.998ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14932025-05-30T21:31:45.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14942025-05-30T21:31:46.000ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
14952025-05-30T21:31:46.000ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14962025-05-30T21:31:46.000ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 60.513585ms iteration = 12 status = {"errors":[],"requests_created_ok":[]}
14972025-05-30T21:31:46.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14982025-05-30T21:31:46.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
14992025-05-30T21:31:46.001ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
15002025-05-30T21:31:46.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
15012025-05-30T21:31:46.001ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15022025-05-30T21:31:46.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15032025-05-30T21:31:46.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15042025-05-30T21:31:46.001ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15052025-05-30T21:31:46.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
15062025-05-30T21:31:46.003ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
15072025-05-30T21:31:46.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
15082025-05-30T21:31:46.006ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15092025-05-30T21:31:46.007ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
15102025-05-30T21:31:46.007ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
15112025-05-30T21:31:46.007ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
15122025-05-30T21:31:46.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
15132025-05-30T21:31:46.009ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
15142025-05-30T21:31:46.009ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15152025-05-30T21:31:46.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15162025-05-30T21:31:46.010ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15172025-05-30T21:31:46.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15182025-05-30T21:31:46.011ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15192025-05-30T21:31:46.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15202025-05-30T21:31:46.014ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
15212025-05-30T21:31:46.014ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
15222025-05-30T21:31:46.014ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "10532cf4-e2d1-43d2-9289-5a0eab8c80c7", "content-length": "26", "date": "Fri, 30 May 2025 21:31:45 GMT"} })
15232025-05-30T21:31:46.014ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15242025-05-30T21:31:46.016ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
15252025-05-30T21:31:46.019ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
15262025-05-30T21:31:46.019ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
15272025-05-30T21:31:46.019ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "73c55996-5e23-4b8e-bc46-75bf859e0226", "content-length": "26", "date": "Fri, 30 May 2025 21:31:45 GMT"} })
15282025-05-30T21:31:46.019ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15292025-05-30T21:31:46.021ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1440, tv_nsec: 959671757 } })
15302025-05-30T21:31:46.021ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
15312025-05-30T21:31:46.021ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
15322025-05-30T21:31:46.035ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 95.581698ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
15332025-05-30T21:31:46.041ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 101.295434ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
15342025-05-30T21:31:46.056ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
15352025-05-30T21:31:46.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15362025-05-30T21:31:46.056ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
15372025-05-30T21:31:46.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
15382025-05-30T21:31:46.058ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15392025-05-30T21:31:46.069ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
15402025-05-30T21:31:46.069ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
15412025-05-30T21:31:46.069ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
15422025-05-30T21:31:46.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
15432025-05-30T21:31:46.070ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
15442025-05-30T21:31:46.070ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
15452025-05-30T21:31:46.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15462025-05-30T21:31:46.072ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15472025-05-30T21:31:46.076ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
15482025-05-30T21:31:46.076ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15492025-05-30T21:31:46.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15502025-05-30T21:31:46.078ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15512025-05-30T21:31:46.083ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
15522025-05-30T21:31:46.083ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15532025-05-30T21:31:46.083ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15542025-05-30T21:31:46.092ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 152.712304ms iteration = 12 status = 0
15552025-05-30T21:31:46.094ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
15562025-05-30T21:31:46.094ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 154.966779ms iteration = 34 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
15572025-05-30T21:31:46.097ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
15582025-05-30T21:31:46.097ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 157.918534ms iteration = 12 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
15592025-05-30T21:31:46.110ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
15602025-05-30T21:31:46.110ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
15612025-05-30T21:31:46.111ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15622025-05-30T21:31:46.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15632025-05-30T21:31:46.112ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
15642025-05-30T21:31:46.112ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
15652025-05-30T21:31:46.113ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e983908-1f27-42e4-bc1c-0e788b08278e", "content-length": "26", "date": "Fri, 30 May 2025 21:31:46 GMT"} })
15662025-05-30T21:31:46.113ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15672025-05-30T21:31:46.115ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
15682025-05-30T21:31:46.119ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 179.463741ms iteration = 12 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
15692025-05-30T21:31:46.123ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 183.225164ms iteration = 12 status = {}
15702025-05-30T21:31:46.129ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
15712025-05-30T21:31:46.129ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 189.915042ms iteration = 12 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
15722025-05-30T21:31:46.145ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
15732025-05-30T21:31:46.145ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
15742025-05-30T21:31:46.145ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 205.315808ms iteration = 12 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
15752025-05-30T21:31:46.170ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 230.211684ms iteration = 13 status = {"physical_disks_added":0}
15762025-05-30T21:31:46.179ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
15772025-05-30T21:31:46.179ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
15782025-05-30T21:31:46.180ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "535203e0-4387-4aa5-98ac-dcef70d72721", "content-length": "372", "date": "Fri, 30 May 2025 21:31:45 GMT"} })
15792025-05-30T21:31:46.180ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15802025-05-30T21:31:46.180ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15812025-05-30T21:31:46.181ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 241.616485ms iteration = 12 status = null
15822025-05-30T21:31:46.231ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
15832025-05-30T21:31:46.231ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
15842025-05-30T21:31:46.231ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "caee3e15-8ba2-4c79-a91d-ba3777faddb8", "date": "Fri, 30 May 2025 21:31:45 GMT"} })
15852025-05-30T21:31:46.231ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15862025-05-30T21:31:46.231ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15872025-05-30T21:31:46.233ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15882025-05-30T21:31:46.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15892025-05-30T21:31:46.237ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15902025-05-30T21:31:46.237ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15912025-05-30T21:31:46.241ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
15922025-05-30T21:31:46.241ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
15932025-05-30T21:31:46.241ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "f7311c02-b368-4511-b3d4-f0ebd027ce00", "date": "Fri, 30 May 2025 21:31:45 GMT"} })
15942025-05-30T21:31:46.243ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 303.486876ms iteration = 12 status = {}
15952025-05-30T21:31:46.244ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15962025-05-30T21:31:46.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15972025-05-30T21:31:46.246ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
15982025-05-30T21:31:46.246ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15992025-05-30T21:31:46.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16002025-05-30T21:31:46.259ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
16012025-05-30T21:31:46.259ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
16022025-05-30T21:31:46.259ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
16032025-05-30T21:31:46.259ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
16042025-05-30T21:31:46.259ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16052025-05-30T21:31:46.259ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16062025-05-30T21:31:46.259ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16072025-05-30T21:31:46.259ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
16082025-05-30T21:31:46.259ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16092025-05-30T21:31:46.259ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
16102025-05-30T21:31:46.259ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
16112025-05-30T21:31:46.259ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16122025-05-30T21:31:46.259ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16132025-05-30T21:31:46.262ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
16142025-05-30T21:31:46.262ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
16152025-05-30T21:31:46.286ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16162025-05-30T21:31:46.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16172025-05-30T21:31:46.319ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16182025-05-30T21:31:46.321ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
16192025-05-30T21:31:46.321ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16202025-05-30T21:31:46.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16212025-05-30T21:31:46.322ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16222025-05-30T21:31:46.324ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
16232025-05-30T21:31:46.324ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16242025-05-30T21:31:46.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16252025-05-30T21:31:46.373ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "4cb5ea32-7c00-4989-a97d-e4799f567a87", "date": "Fri, 30 May 2025 21:31:45 GMT"} })
16262025-05-30T21:31:46.373ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
16272025-05-30T21:31:46.373ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
16282025-05-30T21:31:46.373ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "efaf2e2f-11f6-46fe-b332-a7cbfca70f98", "content-length": "682", "date": "Fri, 30 May 2025 21:31:45 GMT"} })
16292025-05-30T21:31:46.373ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16302025-05-30T21:31:46.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16312025-05-30T21:31:46.378ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16322025-05-30T21:31:46.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16332025-05-30T21:31:46.380ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16342025-05-30T21:31:46.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16352025-05-30T21:31:46.383ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16362025-05-30T21:31:46.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16372025-05-30T21:31:46.390ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 450.740237ms iteration = 12 status = {}
16382025-05-30T21:31:46.428ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
16392025-05-30T21:31:46.482ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a1747ea-c169-43f3-ba01-86096bcf9495", "content-length": "691", "date": "Fri, 30 May 2025 21:31:45 GMT"} })
16402025-05-30T21:31:46.482ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
16412025-05-30T21:31:46.482ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
16422025-05-30T21:31:46.482ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
16432025-05-30T21:31:46.483ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 543.695454ms iteration = 12 status = {}
16442025-05-30T21:31:48.775ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 401 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 776b2305-a249-416b-b2e1-ebaa15cc9620 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
16452025-05-30T21:31:55.939ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
16462025-05-30T21:31:55.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 35 reason = Timeout
16472025-05-30T21:31:55.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16482025-05-30T21:31:55.939ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16492025-05-30T21:31:55.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16502025-05-30T21:31:55.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16512025-05-30T21:31:55.975ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16522025-05-30T21:31:55.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16532025-05-30T21:31:55.978ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16542025-05-30T21:31:55.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
16552025-05-30T21:31:55.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16562025-05-30T21:31:55.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16572025-05-30T21:31:55.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16582025-05-30T21:31:55.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16592025-05-30T21:31:55.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16602025-05-30T21:31:55.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
16612025-05-30T21:31:55.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16622025-05-30T21:31:55.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16632025-05-30T21:31:55.998ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
16642025-05-30T21:31:55.998ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 59.244011ms iteration = 35 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
16652025-05-30T21:31:56.007ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 67.910371ms iteration = 35 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16662025-05-30T21:31:58.765ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35849 req_id = 1ab58c8b-2cc3-41f3-9f1d-87b8cb354f3b roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
16672025-05-30T21:31:58.766ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35849 req_id = 1ab58c8b-2cc3-41f3-9f1d-87b8cb354f3b resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
16682025-05-30T21:31:58.768ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3321 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35849 req_id = 1ab58c8b-2cc3-41f3-9f1d-87b8cb354f3b response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
16692025-05-30T21:31:58.769ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35849 req_id = a836ab1c-fc33-4a7c-8469-953959485c3b roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
16702025-05-30T21:31:58.769ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35849 req_id = a836ab1c-fc33-4a7c-8469-953959485c3b resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
16712025-05-30T21:31:58.771ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2379 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35849 req_id = a836ab1c-fc33-4a7c-8469-953959485c3b response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
16722025-05-30T21:31:58.774ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 350 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 4ec099d4-406c-47c0-baeb-c727121e262f response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
16732025-05-30T21:32:05.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 36 reason = Timeout
16742025-05-30T21:32:05.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
16752025-05-30T21:32:05.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16762025-05-30T21:32:05.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16772025-05-30T21:32:05.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16782025-05-30T21:32:05.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16792025-05-30T21:32:05.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16802025-05-30T21:32:05.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16812025-05-30T21:32:05.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16822025-05-30T21:32:05.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
16832025-05-30T21:32:05.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16842025-05-30T21:32:05.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16852025-05-30T21:32:05.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16862025-05-30T21:32:05.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16872025-05-30T21:32:05.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16882025-05-30T21:32:05.952ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
16892025-05-30T21:32:05.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16902025-05-30T21:32:05.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16912025-05-30T21:32:05.964ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
16922025-05-30T21:32:05.964ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.468461ms iteration = 36 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
16932025-05-30T21:32:05.980ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 40.168061ms iteration = 36 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16942025-05-30T21:32:08.775ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 301 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 270d84c4-bbc3-4138-a673-2e7521d50803 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
16952025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 37 reason = Timeout
16962025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
16972025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
16982025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16992025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
17002025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
17012025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
17022025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17032025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
17042025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 7 reason = Timeout
17052025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
17062025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
17072025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
17082025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
17092025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
17102025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17112025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17122025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
17132025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
17142025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 13 reason = Timeout
17152025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
17162025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 13 reason = Timeout
17172025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
17182025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
17192025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 7 reason = Timeout
17202025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 4 reason = Timeout
17212025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
17222025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
17232025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
17242025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
17252025-05-30T21:32:15.939ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e
17262025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 13 reason = Timeout
17272025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 16.16\u{b5}s iteration = 8 status = {"enabled":false,"target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e"}
17282025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
17292025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 14 reason = Timeout
17302025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
17312025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17322025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 7 reason = Timeout
17332025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
17342025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 7 reason = Timeout
17352025-05-30T21:32:15.939ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
17362025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 7 reason = Timeout
17372025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
17382025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
17392025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 13 reason = Timeout
17402025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
17412025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 13 reason = Timeout
17422025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17432025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 13 reason = Timeout
17442025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 13 reason = Timeout
17452025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
17462025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
17472025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 13 reason = Timeout
17482025-05-30T21:32:15.939ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
17492025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
17502025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17512025-05-30T21:32:15.939ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 7 reason = Timeout
17522025-05-30T21:32:15.940ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-30 21:22:15.939860705 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
17532025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
17542025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 13 reason = Timeout
17552025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
17562025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
17572025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
17582025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17592025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 7 reason = Timeout
17602025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
17612025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 13 reason = Timeout
17622025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
17632025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
17642025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
17652025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 13 reason = Timeout
17662025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17672025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17682025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17692025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
17702025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17712025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
17722025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17732025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17742025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
17752025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
17762025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
17772025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
17782025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17792025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
17802025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
17812025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
17822025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
17832025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
17842025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17852025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
17862025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
17872025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
17882025-05-30T21:32:15.940ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2a161087-f464-41f8-b020-0f9e83fd5351.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1470, tv_nsec: 878746012 } })
17892025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17902025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
17912025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
17922025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17932025-05-30T21:32:15.940ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0413f6b4-86a1-42a1-919a-e8f42ff9f638.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("acb1452d-0a02-4b36-a462-c8ce2f96a488.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:2::1))) }], valid_until: Instant { tv_sec: 1470, tv_nsec: 878772862 } })
17942025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
17952025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17962025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
17972025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
17982025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17992025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18002025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
18012025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18022025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
18032025-05-30T21:32:15.940ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
18042025-05-30T21:32:15.941ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.251744ms iteration = 7 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
18052025-05-30T21:32:15.941ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } dns_group = internal
18062025-05-30T21:32:15.941ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.444676ms iteration = 7 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353"]}
18072025-05-30T21:32:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18082025-05-30T21:32:15.941ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
18092025-05-30T21:32:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18102025-05-30T21:32:15.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18112025-05-30T21:32:15.942ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18122025-05-30T21:32:15.942ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1470, tv_nsec: 881019408 } })
18132025-05-30T21:32:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18142025-05-30T21:32:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18152025-05-30T21:32:15.942ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
18162025-05-30T21:32:15.943ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.238118ms iteration = 13 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
18172025-05-30T21:32:15.944ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18182025-05-30T21:32:15.944ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18192025-05-30T21:32:15.946ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:202 num_receivers = 0
18202025-05-30T21:32:15.946ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 6.954131ms iteration = 7 status = {"by_rx":{},"error":null}
18212025-05-30T21:32:15.947ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
18222025-05-30T21:32:15.947ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18232025-05-30T21:32:15.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18242025-05-30T21:32:15.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
18252025-05-30T21:32:15.948ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18262025-05-30T21:32:15.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18272025-05-30T21:32:15.950ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18282025-05-30T21:32:15.950ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1470, tv_nsec: 888762537 } })
18292025-05-30T21:32:15.950ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
18302025-05-30T21:32:15.952ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
18312025-05-30T21:32:15.952ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18322025-05-30T21:32:15.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18332025-05-30T21:32:15.953ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18342025-05-30T21:32:15.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
18352025-05-30T21:32:15.955ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
18362025-05-30T21:32:15.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
18372025-05-30T21:32:15.958ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 18.385572ms iteration = 7 status = {"errors":[],"found_port_ok":[]}
18382025-05-30T21:32:15.958ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18392025-05-30T21:32:15.959ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18402025-05-30T21:32:15.960ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
18412025-05-30T21:32:15.960ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18422025-05-30T21:32:15.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18432025-05-30T21:32:15.960ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18442025-05-30T21:32:15.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18452025-05-30T21:32:15.962ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-30 21:22:15.939860705 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
18462025-05-30T21:32:15.962ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 22.997024ms iteration = 7 status = {"expiration":"2025-05-30T21:22:15.939860705Z","pruned":[]}
18472025-05-30T21:32:15.963ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18482025-05-30T21:32:15.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18492025-05-30T21:32:15.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18502025-05-30T21:32:15.966ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18512025-05-30T21:32:15.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
18522025-05-30T21:32:15.968ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
18532025-05-30T21:32:15.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18542025-05-30T21:32:15.970ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18552025-05-30T21:32:15.974ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
18562025-05-30T21:32:15.974ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
18572025-05-30T21:32:15.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18582025-05-30T21:32:15.976ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 36.767442ms iteration = 37 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
18592025-05-30T21:32:15.978ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18602025-05-30T21:32:15.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
18612025-05-30T21:32:15.984ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18622025-05-30T21:32:15.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
18632025-05-30T21:32:15.986ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18642025-05-30T21:32:15.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18652025-05-30T21:32:15.987ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18662025-05-30T21:32:15.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18672025-05-30T21:32:15.988ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
18682025-05-30T21:32:15.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
18692025-05-30T21:32:15.989ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18702025-05-30T21:32:15.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18712025-05-30T21:32:15.990ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
18722025-05-30T21:32:15.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
18732025-05-30T21:32:15.992ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
18742025-05-30T21:32:15.992ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
18752025-05-30T21:32:15.992ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
18762025-05-30T21:32:15.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
18772025-05-30T21:32:15.995ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
18782025-05-30T21:32:15.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
18792025-05-30T21:32:15.996ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18802025-05-30T21:32:15.998ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18812025-05-30T21:32:15.998ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18822025-05-30T21:32:15.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18832025-05-30T21:32:15.998ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18842025-05-30T21:32:15.999ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
18852025-05-30T21:32:16.000ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18862025-05-30T21:32:16.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18872025-05-30T21:32:16.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
18882025-05-30T21:32:16.001ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
18892025-05-30T21:32:16.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
18902025-05-30T21:32:16.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
18912025-05-30T21:32:16.002ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
18922025-05-30T21:32:16.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18932025-05-30T21:32:16.002ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
18942025-05-30T21:32:16.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
18952025-05-30T21:32:16.006ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 66.324632ms iteration = 13 status = {"errors":[],"finish_invoked_ok":[]}
18962025-05-30T21:32:16.007ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 67.593636ms iteration = 13 status = {"errors":[],"garbage_collect_requested":[]}
18972025-05-30T21:32:16.008ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18982025-05-30T21:32:16.009ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 69.791212ms iteration = 7 status = {"dispatched":[],"errors":[],"glob_version":"144.0.0","globs_reprocessed":{},"no_receivers":[]}
18992025-05-30T21:32:16.012ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
19002025-05-30T21:32:16.012ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19012025-05-30T21:32:16.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19022025-05-30T21:32:16.013ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 74.04169ms iteration = 13 status = {"errors":[],"requests_created_ok":[]}
19032025-05-30T21:32:16.014ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19042025-05-30T21:32:16.016ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
19052025-05-30T21:32:16.016ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
19062025-05-30T21:32:16.016ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 76.512879ms iteration = 7 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
19072025-05-30T21:32:16.018ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19082025-05-30T21:32:16.019ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19092025-05-30T21:32:16.019ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
19102025-05-30T21:32:16.019ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19112025-05-30T21:32:16.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19122025-05-30T21:32:16.020ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19132025-05-30T21:32:16.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
19142025-05-30T21:32:16.021ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19152025-05-30T21:32:16.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19162025-05-30T21:32:16.022ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
19172025-05-30T21:32:16.022ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19182025-05-30T21:32:16.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19192025-05-30T21:32:16.023ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
19202025-05-30T21:32:16.023ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
19212025-05-30T21:32:16.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
19222025-05-30T21:32:16.023ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19232025-05-30T21:32:16.027ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
19242025-05-30T21:32:16.027ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
19252025-05-30T21:32:16.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
19262025-05-30T21:32:16.031ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1470, tv_nsec: 970267503 } })
19272025-05-30T21:32:16.032ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
19282025-05-30T21:32:16.035ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 96.097053ms iteration = 4 status = {"nsuccess":5}
19292025-05-30T21:32:16.038ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
19302025-05-30T21:32:16.038ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
19312025-05-30T21:32:16.038ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
19322025-05-30T21:32:16.038ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19332025-05-30T21:32:16.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19342025-05-30T21:32:16.040ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
19352025-05-30T21:32:16.040ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
19362025-05-30T21:32:16.040ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 100.556325ms iteration = 7 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
19372025-05-30T21:32:16.040ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
19382025-05-30T21:32:16.040ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 100.482094ms iteration = 37 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
19392025-05-30T21:32:16.040ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19402025-05-30T21:32:16.045ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19412025-05-30T21:32:16.045ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19422025-05-30T21:32:16.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19432025-05-30T21:32:16.046ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19442025-05-30T21:32:16.049ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19452025-05-30T21:32:16.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19462025-05-30T21:32:16.051ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19472025-05-30T21:32:16.052ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19482025-05-30T21:32:16.052ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19492025-05-30T21:32:16.053ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19502025-05-30T21:32:16.055ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
19512025-05-30T21:32:16.055ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 115.907991ms iteration = 13 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
19522025-05-30T21:32:16.057ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
19532025-05-30T21:32:16.057ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19542025-05-30T21:32:16.057ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19552025-05-30T21:32:16.061ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 122.044341ms iteration = 13 status = 0
19562025-05-30T21:32:16.063ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
19572025-05-30T21:32:16.063ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
19582025-05-30T21:32:16.063ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
19592025-05-30T21:32:16.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
19602025-05-30T21:32:16.066ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19612025-05-30T21:32:16.067ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 128.290244ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
19622025-05-30T21:32:16.070ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
19632025-05-30T21:32:16.070ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
19642025-05-30T21:32:16.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
19652025-05-30T21:32:16.074ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 135.288984ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
19662025-05-30T21:32:16.079ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
19672025-05-30T21:32:16.081ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
19682025-05-30T21:32:16.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
19692025-05-30T21:32:16.082ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
19702025-05-30T21:32:16.086ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 146.478912ms iteration = 13 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
19712025-05-30T21:32:16.090ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19722025-05-30T21:32:16.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19732025-05-30T21:32:16.091ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19742025-05-30T21:32:16.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19752025-05-30T21:32:16.093ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
19762025-05-30T21:32:16.093ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19772025-05-30T21:32:16.093ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19782025-05-30T21:32:16.093ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb3a7fc8-aec1-4ce7-bdbc-9023e2f91112", "content-length": "26", "date": "Fri, 30 May 2025 21:32:16 GMT"} })
19792025-05-30T21:32:16.093ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19802025-05-30T21:32:16.095ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
19812025-05-30T21:32:16.095ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19822025-05-30T21:32:16.096ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19832025-05-30T21:32:16.096ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
19842025-05-30T21:32:16.098ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
19852025-05-30T21:32:16.098ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 158.155437ms iteration = 13 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
19862025-05-30T21:32:16.098ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
19872025-05-30T21:32:16.098ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19882025-05-30T21:32:16.098ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5031739-d841-4039-b755-a072ccc325f8", "content-length": "26", "date": "Fri, 30 May 2025 21:32:16 GMT"} })
19892025-05-30T21:32:16.098ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19902025-05-30T21:32:16.101ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
19912025-05-30T21:32:16.107ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
19922025-05-30T21:32:16.107ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
19932025-05-30T21:32:16.107ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19942025-05-30T21:32:16.108ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19952025-05-30T21:32:16.110ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19962025-05-30T21:32:16.113ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19972025-05-30T21:32:16.113ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19982025-05-30T21:32:16.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19992025-05-30T21:32:16.117ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
20002025-05-30T21:32:16.117ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
20012025-05-30T21:32:16.117ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-30 21:26:17.226115 UTC dns_group = internal generation = 1
20022025-05-30T21:32:16.117ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 178.278528ms iteration = 7 status = {"generation":1}
20032025-05-30T21:32:16.129ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
20042025-05-30T21:32:16.129ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
20052025-05-30T21:32:16.129ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
20062025-05-30T21:32:16.130ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20072025-05-30T21:32:16.130ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
20082025-05-30T21:32:16.130ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
20092025-05-30T21:32:16.130ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 190.529369ms iteration = 13 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
20102025-05-30T21:32:16.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20112025-05-30T21:32:16.133ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20122025-05-30T21:32:16.137ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20132025-05-30T21:32:16.137ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20142025-05-30T21:32:16.137ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20152025-05-30T21:32:16.138ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20162025-05-30T21:32:16.140ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
20172025-05-30T21:32:16.140ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 0 version = 2
20182025-05-30T21:32:16.140ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20192025-05-30T21:32:16.140ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20202025-05-30T21:32:16.140ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20212025-05-30T21:32:16.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20222025-05-30T21:32:16.140ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 200.824717ms iteration = 14 status = {"physical_disks_added":0}
20232025-05-30T21:32:16.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20242025-05-30T21:32:16.140ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
20252025-05-30T21:32:16.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
20262025-05-30T21:32:16.168ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20272025-05-30T21:32:16.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20282025-05-30T21:32:16.253ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
20292025-05-30T21:32:16.259ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 320.37605ms iteration = 13 status = {}
20302025-05-30T21:32:16.266ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
20312025-05-30T21:32:16.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
20322025-05-30T21:32:16.301ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
20332025-05-30T21:32:16.301ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
20342025-05-30T21:32:16.302ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5a0a5c4-3b99-42d5-a54c-a93372bf4dee", "content-length": "372", "date": "Fri, 30 May 2025 21:32:15 GMT"} })
20352025-05-30T21:32:16.302ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20362025-05-30T21:32:16.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20372025-05-30T21:32:16.304ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20382025-05-30T21:32:16.305ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20392025-05-30T21:32:16.306ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
20402025-05-30T21:32:16.306ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
20412025-05-30T21:32:16.306ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 366.51557ms iteration = 13 status = null
20422025-05-30T21:32:16.306ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd8aae21-9195-4067-9a74-7a990d2a8aa7", "content-length": "26", "date": "Fri, 30 May 2025 21:32:16 GMT"} })
20432025-05-30T21:32:16.306ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
20442025-05-30T21:32:16.307ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
20452025-05-30T21:32:16.307ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20462025-05-30T21:32:16.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20472025-05-30T21:32:16.308ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20482025-05-30T21:32:16.308ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20492025-05-30T21:32:16.308ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
20502025-05-30T21:32:16.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20512025-05-30T21:32:16.310ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20522025-05-30T21:32:16.311ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 1 version = 2
20532025-05-30T21:32:16.311ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
20542025-05-30T21:32:16.311ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-30 21:26:18.957109 UTC dns_group = external generation = 2
20552025-05-30T21:32:16.311ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 371.7739ms iteration = 7 status = {"generation":2}
20562025-05-30T21:32:16.312ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
20572025-05-30T21:32:16.312ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20582025-05-30T21:32:16.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20592025-05-30T21:32:16.336ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
20602025-05-30T21:32:16.336ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "e7191649-9cfd-41e2-adb8-41378cc4a198", "date": "Fri, 30 May 2025 21:32:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
20612025-05-30T21:32:16.336ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
20622025-05-30T21:32:16.339ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "b3c2c3f2-3719-4034-bb5e-16619ad22ba7", "date": "Fri, 30 May 2025 21:32:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
20632025-05-30T21:32:16.339ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
20642025-05-30T21:32:16.341ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
20652025-05-30T21:32:16.341ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 401.585693ms iteration = 9 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
20662025-05-30T21:32:16.368ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-30T21:26:13.986396Z, time_modified: 2025-05-30T21:26:13.986396Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }) }
20672025-05-30T21:32:16.368ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 429.20453ms iteration = 7 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]}},"default_endpoint":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
20682025-05-30T21:32:16.372ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
20692025-05-30T21:32:16.372ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "6e868b39-1c47-4cac-97a9-84f12b20a61d", "date": "Fri, 30 May 2025 21:32:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
20702025-05-30T21:32:16.372ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
20712025-05-30T21:32:16.374ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "67790d5f-e4c7-47e0-8572-e165e685f57d", "date": "Fri, 30 May 2025 21:32:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
20722025-05-30T21:32:16.374ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
20732025-05-30T21:32:16.376ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "dc4820d5-24cd-4000-8674-c0c98e853d9c", "date": "Fri, 30 May 2025 21:32:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
20742025-05-30T21:32:16.376ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
20752025-05-30T21:32:16.378ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
20762025-05-30T21:32:16.378ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 438.77533ms iteration = 9 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
20772025-05-30T21:32:16.383ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
20782025-05-30T21:32:16.383ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
20792025-05-30T21:32:16.384ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "024eb8b2-6183-4f39-8c61-070b31cb3278", "date": "Fri, 30 May 2025 21:32:15 GMT"} })
20802025-05-30T21:32:16.384ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20812025-05-30T21:32:16.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20822025-05-30T21:32:16.386ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20832025-05-30T21:32:16.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20842025-05-30T21:32:16.393ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20852025-05-30T21:32:16.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20862025-05-30T21:32:16.398ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
20872025-05-30T21:32:16.398ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
20882025-05-30T21:32:16.398ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "17a6751b-adb7-4113-9470-98ca01e763f5", "date": "Fri, 30 May 2025 21:32:15 GMT"} })
20892025-05-30T21:32:16.398ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
20902025-05-30T21:32:16.399ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee8e36a3-8037-4463-aacf-32bff0445f7a", "content-length": "691", "date": "Fri, 30 May 2025 21:32:15 GMT"} })
20912025-05-30T21:32:16.399ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
20922025-05-30T21:32:16.399ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
20932025-05-30T21:32:16.399ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
20942025-05-30T21:32:16.400ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 460.460359ms iteration = 13 status = {}
20952025-05-30T21:32:16.401ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 461.563391ms iteration = 13 status = {}
20962025-05-30T21:32:16.464ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20972025-05-30T21:32:16.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20982025-05-30T21:32:16.466ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
20992025-05-30T21:32:16.466ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21002025-05-30T21:32:16.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21012025-05-30T21:32:16.469ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
21022025-05-30T21:32:16.469ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 529.991277ms iteration = 7 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
21032025-05-30T21:32:16.479ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
21042025-05-30T21:32:16.479ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
21052025-05-30T21:32:16.479ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
21062025-05-30T21:32:16.479ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
21072025-05-30T21:32:16.479ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
21082025-05-30T21:32:16.479ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
21092025-05-30T21:32:16.479ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
21102025-05-30T21:32:16.479ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
21112025-05-30T21:32:16.479ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
21122025-05-30T21:32:16.479ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
21132025-05-30T21:32:16.479ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
21142025-05-30T21:32:16.479ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21152025-05-30T21:32:16.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21162025-05-30T21:32:16.481ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
21172025-05-30T21:32:16.481ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
21182025-05-30T21:32:16.519ZDEBGnexus (producer-registration-task): registering as oximeter producer with Nexus registration_count = 2
21192025-05-30T21:32:16.519ZDEBGnexus (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221
21202025-05-30T21:32:16.549ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
21212025-05-30T21:32:16.549ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:61992
21222025-05-30T21:32:16.550ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:61992 req_id = 81a93ac9-0a6b-4e62-aec6-71b41c900395 roles = RoleSet { roles: {} } uri = /metrics/producers
21232025-05-30T21:32:16.550ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:61992 req_id = 81a93ac9-0a6b-4e62-aec6-71b41c900395 resource = Database result = Ok(()) uri = /metrics/producers
21242025-05-30T21:32:16.587ZINFOnexus: assigned collector to new producer collector_id = bc707af4-d9a3-402f-b2b6-642b29342fa8 file = nexus/src/app/oximeter.rs:76 producer_id = bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
21252025-05-30T21:32:16.587ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 37967 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:61992 req_id = 81a93ac9-0a6b-4e62-aec6-71b41c900395 response_code = 201 uri = /metrics/producers
21262025-05-30T21:32:16.588ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "81a93ac9-0a6b-4e62-aec6-71b41c900395", "content-length": "41", "date": "Fri, 30 May 2025 21:32:16 GMT"} })
21272025-05-30T21:32:16.588ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "b1d0edea-5af3-4c56-a935-b5081c997884", "date": "Fri, 30 May 2025 21:32:15 GMT"} })
21282025-05-30T21:32:16.588ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
21292025-05-30T21:32:16.588ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
21302025-05-30T21:32:16.588ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4cad8b3-31d0-4736-9b53-b579fcfc534c", "content-length": "682", "date": "Fri, 30 May 2025 21:32:16 GMT"} })
21312025-05-30T21:32:16.588ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21322025-05-30T21:32:16.589ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21332025-05-30T21:32:16.589ZDEBGnexus (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
21342025-05-30T21:32:16.592ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21352025-05-30T21:32:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21362025-05-30T21:32:16.596ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21372025-05-30T21:32:16.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21382025-05-30T21:32:16.598ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21392025-05-30T21:32:16.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21402025-05-30T21:32:16.605ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 666.08134ms iteration = 13 status = {}
21412025-05-30T21:32:18.775ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 328 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 44c9f0bb-4433-4296-8647-f6a908069d72 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
21422025-05-30T21:32:25.939ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
21432025-05-30T21:32:25.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 38 reason = Timeout
21442025-05-30T21:32:25.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21452025-05-30T21:32:25.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21462025-05-30T21:32:25.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21472025-05-30T21:32:25.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21482025-05-30T21:32:25.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21492025-05-30T21:32:25.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21502025-05-30T21:32:25.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21512025-05-30T21:32:25.944ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.34359ms iteration = 38 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
21522025-05-30T21:32:25.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21532025-05-30T21:32:25.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21542025-05-30T21:32:25.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21552025-05-30T21:32:25.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21562025-05-30T21:32:25.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21572025-05-30T21:32:25.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21582025-05-30T21:32:25.950ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
21592025-05-30T21:32:25.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21602025-05-30T21:32:25.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21612025-05-30T21:32:25.961ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
21622025-05-30T21:32:25.961ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.240054ms iteration = 38 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
21632025-05-30T21:32:28.775ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 352 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = e498125d-4e58-4913-ac78-90f0075958ed response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
21642025-05-30T21:32:35.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
21652025-05-30T21:32:35.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 39 reason = Timeout
21662025-05-30T21:32:35.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21672025-05-30T21:32:35.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21682025-05-30T21:32:35.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21692025-05-30T21:32:35.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21702025-05-30T21:32:35.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21712025-05-30T21:32:35.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21722025-05-30T21:32:35.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21732025-05-30T21:32:35.944ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.205049ms iteration = 39 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
21742025-05-30T21:32:35.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21752025-05-30T21:32:35.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21762025-05-30T21:32:35.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21772025-05-30T21:32:35.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21782025-05-30T21:32:35.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21792025-05-30T21:32:35.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21802025-05-30T21:32:35.952ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
21812025-05-30T21:32:35.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21822025-05-30T21:32:35.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21832025-05-30T21:32:35.963ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
21842025-05-30T21:32:35.963ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.461079ms iteration = 39 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
21852025-05-30T21:32:38.775ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 346 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = d301bf13-1115-4701-b180-14d5ed93e8b1 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
21862025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
21872025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 14 reason = Timeout
21882025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 14 reason = Timeout
21892025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21902025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
21912025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
21922025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
21932025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
21942025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
21952025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21962025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 14 reason = Timeout
21972025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 14 reason = Timeout
21982025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
21992025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
22002025-05-30T21:32:45.939ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
22012025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
22022025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
22032025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 14 reason = Timeout
22042025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 14 reason = Timeout
22052025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22062025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
22072025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 14 reason = Timeout
22082025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
22092025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 14 reason = Timeout
22102025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
22112025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
22122025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 14 reason = Timeout
22132025-05-30T21:32:45.940ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
22142025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 15 reason = Timeout
22152025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22162025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22172025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
22182025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
22192025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 14 reason = Timeout
22202025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22212025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 40 reason = Timeout
22222025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22232025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 14 reason = Timeout
22242025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22252025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
22262025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
22272025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22282025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22292025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
22302025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
22312025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22322025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
22332025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
22342025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22352025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22362025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
22372025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22382025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
22392025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22402025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
22412025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
22422025-05-30T21:32:45.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22432025-05-30T21:32:45.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22442025-05-30T21:32:45.943ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.086517ms iteration = 14 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
22452025-05-30T21:32:45.944ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1500, tv_nsec: 882562338 } })
22462025-05-30T21:32:45.944ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22472025-05-30T21:32:45.944ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
22482025-05-30T21:32:45.945ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22492025-05-30T21:32:45.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
22502025-05-30T21:32:45.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22512025-05-30T21:32:45.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22522025-05-30T21:32:45.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
22532025-05-30T21:32:45.948ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22542025-05-30T21:32:45.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
22552025-05-30T21:32:45.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22562025-05-30T21:32:45.948ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22572025-05-30T21:32:45.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22582025-05-30T21:32:45.948ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22592025-05-30T21:32:45.948ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22602025-05-30T21:32:45.951ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
22612025-05-30T21:32:45.951ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22622025-05-30T21:32:45.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22632025-05-30T21:32:45.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
22642025-05-30T21:32:45.951ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
22652025-05-30T21:32:45.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
22662025-05-30T21:32:45.953ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
22672025-05-30T21:32:45.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
22682025-05-30T21:32:45.956ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22692025-05-30T21:32:45.956ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 16.623791ms iteration = 14 status = {"errors":[],"finish_invoked_ok":[]}
22702025-05-30T21:32:45.957ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22712025-05-30T21:32:45.957ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1500, tv_nsec: 895863140 } })
22722025-05-30T21:32:45.958ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
22732025-05-30T21:32:45.959ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
22742025-05-30T21:32:45.959ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22752025-05-30T21:32:45.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22762025-05-30T21:32:45.959ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22772025-05-30T21:32:45.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22782025-05-30T21:32:45.961ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
22792025-05-30T21:32:45.961ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22802025-05-30T21:32:45.961ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22812025-05-30T21:32:45.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22822025-05-30T21:32:45.961ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22832025-05-30T21:32:45.961ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1500, tv_nsec: 899622563 } })
22842025-05-30T21:32:45.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22852025-05-30T21:32:45.961ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
22862025-05-30T21:32:45.963ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
22872025-05-30T21:32:45.963ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22882025-05-30T21:32:45.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22892025-05-30T21:32:45.963ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22902025-05-30T21:32:45.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22912025-05-30T21:32:45.966ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22922025-05-30T21:32:45.967ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
22932025-05-30T21:32:45.967ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 27.459556ms iteration = 14 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
22942025-05-30T21:32:45.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
22952025-05-30T21:32:45.968ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22962025-05-30T21:32:45.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22972025-05-30T21:32:45.988ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
22982025-05-30T21:32:45.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
22992025-05-30T21:32:45.993ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
23002025-05-30T21:32:45.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
23012025-05-30T21:32:45.995ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
23022025-05-30T21:32:45.995ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
23032025-05-30T21:32:45.995ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
23042025-05-30T21:32:45.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
23052025-05-30T21:32:45.996ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 56.493699ms iteration = 14 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
23062025-05-30T21:32:45.998ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
23072025-05-30T21:32:45.998ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
23082025-05-30T21:32:45.998ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
23092025-05-30T21:32:45.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
23102025-05-30T21:32:45.999ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 59.1968ms iteration = 14 status = {"errors":[],"requests_created_ok":[]}
23112025-05-30T21:32:46.002ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
23122025-05-30T21:32:46.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
23132025-05-30T21:32:46.003ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
23142025-05-30T21:32:46.003ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23152025-05-30T21:32:46.003ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "df4680f1-29e8-4157-93db-4c0463a91691", "content-length": "26", "date": "Fri, 30 May 2025 21:32:45 GMT"} })
23162025-05-30T21:32:46.003ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23172025-05-30T21:32:46.003ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23182025-05-30T21:32:46.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23192025-05-30T21:32:46.005ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 65.654194ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
23202025-05-30T21:32:46.006ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
23212025-05-30T21:32:46.006ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23222025-05-30T21:32:46.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23232025-05-30T21:32:46.010ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23242025-05-30T21:32:46.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23252025-05-30T21:32:46.014ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 74.065621ms iteration = 14 status = {"errors":[],"garbage_collect_requested":[]}
23262025-05-30T21:32:46.015ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23272025-05-30T21:32:46.016ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23282025-05-30T21:32:46.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23292025-05-30T21:32:46.017ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23302025-05-30T21:32:46.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23312025-05-30T21:32:46.019ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
23322025-05-30T21:32:46.019ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23332025-05-30T21:32:46.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23342025-05-30T21:32:46.023ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23352025-05-30T21:32:46.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23362025-05-30T21:32:46.024ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23372025-05-30T21:32:46.024ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23382025-05-30T21:32:46.026ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
23392025-05-30T21:32:46.026ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23402025-05-30T21:32:46.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23412025-05-30T21:32:46.027ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
23422025-05-30T21:32:46.027ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23432025-05-30T21:32:46.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23442025-05-30T21:32:46.028ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23452025-05-30T21:32:46.029ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
23462025-05-30T21:32:46.029ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23472025-05-30T21:32:46.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23482025-05-30T21:32:46.031ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 91.548221ms iteration = 14 status = 0
23492025-05-30T21:32:46.034ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
23502025-05-30T21:32:46.034ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 94.809549ms iteration = 40 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
23512025-05-30T21:32:46.036ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
23522025-05-30T21:32:46.036ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23532025-05-30T21:32:46.036ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6b5aa3c-827f-4a31-af70-bc9391aeff18", "content-length": "26", "date": "Fri, 30 May 2025 21:32:45 GMT"} })
23542025-05-30T21:32:46.036ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23552025-05-30T21:32:46.038ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
23562025-05-30T21:32:46.038ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
23572025-05-30T21:32:46.038ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
23582025-05-30T21:32:46.038ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 98.921236ms iteration = 14 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
23592025-05-30T21:32:46.046ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23602025-05-30T21:32:46.049ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
23612025-05-30T21:32:46.049ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23622025-05-30T21:32:46.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23632025-05-30T21:32:46.050ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 110.207656ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
23642025-05-30T21:32:46.054ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 114.876999ms iteration = 40 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
23652025-05-30T21:32:46.107ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
23662025-05-30T21:32:46.107ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 167.392062ms iteration = 14 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
23672025-05-30T21:32:46.122ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 182.441946ms iteration = 15 status = {"physical_disks_added":0}
23682025-05-30T21:32:46.162ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
23692025-05-30T21:32:46.162ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23702025-05-30T21:32:46.162ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0688a897-3aa0-484f-b1bc-517ad486f833", "content-length": "26", "date": "Fri, 30 May 2025 21:32:46 GMT"} })
23712025-05-30T21:32:46.162ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23722025-05-30T21:32:46.164ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
23732025-05-30T21:32:46.183ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
23742025-05-30T21:32:46.183ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
23752025-05-30T21:32:46.183ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "eca18b62-c494-4937-b72d-a293272549d2", "content-length": "372", "date": "Fri, 30 May 2025 21:32:45 GMT"} })
23762025-05-30T21:32:46.183ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23772025-05-30T21:32:46.184ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23782025-05-30T21:32:46.188ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 248.882718ms iteration = 14 status = {}
23792025-05-30T21:32:46.197ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
23802025-05-30T21:32:46.197ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
23812025-05-30T21:32:46.226ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
23822025-05-30T21:32:46.227ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "220e52cc-26e8-4e04-af36-f1a325ac2f30", "content-length": "691", "date": "Fri, 30 May 2025 21:32:45 GMT"} })
23832025-05-30T21:32:46.227ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
23842025-05-30T21:32:46.227ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
23852025-05-30T21:32:46.227ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
23862025-05-30T21:32:46.229ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 289.356913ms iteration = 14 status = {}
23872025-05-30T21:32:46.229ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 289.400144ms iteration = 14 status = null
23882025-05-30T21:32:46.277ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
23892025-05-30T21:32:46.277ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
23902025-05-30T21:32:46.277ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "53706df6-0d6d-4b27-9c7c-60af0611557b", "date": "Fri, 30 May 2025 21:32:45 GMT"} })
23912025-05-30T21:32:46.277ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23922025-05-30T21:32:46.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23932025-05-30T21:32:46.280ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23942025-05-30T21:32:46.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23952025-05-30T21:32:46.283ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23962025-05-30T21:32:46.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23972025-05-30T21:32:46.286ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
23982025-05-30T21:32:46.286ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
23992025-05-30T21:32:46.287ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "872e9b5c-780e-4e81-b2b4-e08f79769284", "date": "Fri, 30 May 2025 21:32:45 GMT"} })
24002025-05-30T21:32:46.288ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 348.660015ms iteration = 14 status = {}
24012025-05-30T21:32:46.295ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24022025-05-30T21:32:46.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24032025-05-30T21:32:46.297ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
24042025-05-30T21:32:46.297ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24052025-05-30T21:32:46.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24062025-05-30T21:32:46.310ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
24072025-05-30T21:32:46.310ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
24082025-05-30T21:32:46.310ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
24092025-05-30T21:32:46.310ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
24102025-05-30T21:32:46.310ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24112025-05-30T21:32:46.310ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24122025-05-30T21:32:46.310ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24132025-05-30T21:32:46.310ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
24142025-05-30T21:32:46.310ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24152025-05-30T21:32:46.311ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
24162025-05-30T21:32:46.311ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
24172025-05-30T21:32:46.311ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24182025-05-30T21:32:46.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24192025-05-30T21:32:46.313ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
24202025-05-30T21:32:46.314ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
24212025-05-30T21:32:46.404ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "bfef660a-c96f-41b0-ac73-9d2109f0b7fb", "date": "Fri, 30 May 2025 21:32:45 GMT"} })
24222025-05-30T21:32:46.404ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
24232025-05-30T21:32:46.404ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
24242025-05-30T21:32:46.404ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "b527c0b3-f818-4853-8635-8d2d77c94633", "content-length": "682", "date": "Fri, 30 May 2025 21:32:45 GMT"} })
24252025-05-30T21:32:46.404ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24262025-05-30T21:32:46.405ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24272025-05-30T21:32:46.407ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24282025-05-30T21:32:46.407ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24292025-05-30T21:32:46.411ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24302025-05-30T21:32:46.411ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24312025-05-30T21:32:46.413ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24322025-05-30T21:32:46.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24332025-05-30T21:32:46.420ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 480.137864ms iteration = 14 status = {}
24342025-05-30T21:32:48.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 359 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = a45a1aad-5e83-4c26-a1f2-eeafa6d557f0 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
24352025-05-30T21:32:55.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 41 reason = Timeout
24362025-05-30T21:32:55.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24372025-05-30T21:32:55.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
24382025-05-30T21:32:55.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24392025-05-30T21:32:55.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24402025-05-30T21:32:55.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24412025-05-30T21:32:55.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24422025-05-30T21:32:55.945ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24432025-05-30T21:32:55.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24442025-05-30T21:32:55.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24452025-05-30T21:32:55.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24462025-05-30T21:32:55.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24472025-05-30T21:32:55.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24482025-05-30T21:32:55.952ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
24492025-05-30T21:32:55.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24502025-05-30T21:32:55.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24512025-05-30T21:32:55.963ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
24522025-05-30T21:32:55.963ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.329856ms iteration = 41 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
24532025-05-30T21:32:55.971ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24542025-05-30T21:32:55.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24552025-05-30T21:32:56.002ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 61.650818ms iteration = 41 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
24562025-05-30T21:32:58.765ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:63692
24572025-05-30T21:32:58.765ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63692 req_id = 74c16b4f-170b-4600-8b0f-ab92c25abe56 roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
24582025-05-30T21:32:58.766ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63692 req_id = 74c16b4f-170b-4600-8b0f-ab92c25abe56 resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
24592025-05-30T21:32:58.768ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2539 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63692 req_id = 74c16b4f-170b-4600-8b0f-ab92c25abe56 response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
24602025-05-30T21:32:58.768ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63692 req_id = 043bd60f-2953-4297-8f01-15ec15ee8832 roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
24612025-05-30T21:32:58.768ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63692 req_id = 043bd60f-2953-4297-8f01-15ec15ee8832 resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
24622025-05-30T21:32:58.770ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 1907 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63692 req_id = 043bd60f-2953-4297-8f01-15ec15ee8832 response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
24632025-05-30T21:32:58.775ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 292 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 32a871db-9cb9-481e-a7c8-f2201920874e response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
24642025-05-30T21:33:05.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 42 reason = Timeout
24652025-05-30T21:33:05.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
24662025-05-30T21:33:05.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24672025-05-30T21:33:05.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24682025-05-30T21:33:05.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24692025-05-30T21:33:05.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24702025-05-30T21:33:05.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24712025-05-30T21:33:05.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24722025-05-30T21:33:05.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24732025-05-30T21:33:05.945ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.555682ms iteration = 42 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
24742025-05-30T21:33:05.945ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24752025-05-30T21:33:05.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24762025-05-30T21:33:05.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24772025-05-30T21:33:05.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24782025-05-30T21:33:05.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24792025-05-30T21:33:05.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24802025-05-30T21:33:05.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
24812025-05-30T21:33:05.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24822025-05-30T21:33:05.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24832025-05-30T21:33:05.962ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
24842025-05-30T21:33:05.962ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.79212ms iteration = 42 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
24852025-05-30T21:33:08.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 308 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 2cdd3ce1-d41c-4283-9af5-7e17f74449ed response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
24862025-05-30T21:33:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
24872025-05-30T21:33:15.939ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
24882025-05-30T21:33:15.939ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
24892025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
24902025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
24912025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24922025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 15 reason = Timeout
24932025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 8 reason = Timeout
24942025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
24952025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
24962025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 15 reason = Timeout
24972025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 15 reason = Timeout
24982025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
24992025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25002025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 15 reason = Timeout
25012025-05-30T21:33:15.940ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
25022025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 15 reason = Timeout
25032025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25042025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 16 reason = Timeout
25052025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25062025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 43 reason = Timeout
25072025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
25082025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25092025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 8 reason = Timeout
25102025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
25112025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 8 reason = Timeout
25122025-05-30T21:33:15.940ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-30 21:23:15.940350943 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
25132025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
25142025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25152025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 8 reason = Timeout
25162025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
25172025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 8 reason = Timeout
25182025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
25192025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
25202025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
25212025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 8 reason = Timeout
25222025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
25232025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
25242025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
25252025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
25262025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
25272025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 15 reason = Timeout
25282025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
25292025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
25302025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
25312025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 15 reason = Timeout
25322025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
25332025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 15 reason = Timeout
25342025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
25352025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
25362025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25372025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 15 reason = Timeout
25382025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
25392025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 15 reason = Timeout
25402025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
25412025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
25422025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
25432025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
25442025-05-30T21:33:15.940ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e
25452025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
25462025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.38\u{b5}s iteration = 9 status = {"enabled":false,"target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e"}
25472025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
25482025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
25492025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
25502025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
25512025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
25522025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
25532025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 8 reason = Timeout
25542025-05-30T21:33:15.940ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
25552025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 15 reason = Timeout
25562025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
25572025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
25582025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
25592025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
25602025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
25612025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
25622025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
25632025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25642025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
25652025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25662025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25672025-05-30T21:33:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
25682025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
25692025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
25702025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
25712025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
25722025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
25732025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
25742025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
25752025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
25762025-05-30T21:33:15.941ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0413f6b4-86a1-42a1-919a-e8f42ff9f638.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("acb1452d-0a02-4b36-a462-c8ce2f96a488.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:2::1))) }], valid_until: Instant { tv_sec: 1530, tv_nsec: 878907961 } })
25772025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
25782025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
25792025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
25802025-05-30T21:33:15.941ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2a161087-f464-41f8-b020-0f9e83fd5351.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1530, tv_nsec: 878928682 } })
25812025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25822025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
25832025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
25842025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
25852025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
25862025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
25872025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
25882025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
25892025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.036742ms iteration = 8 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
25902025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } dns_group = internal
25912025-05-30T21:33:15.941ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.121612ms iteration = 8 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353"]}
25922025-05-30T21:33:15.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25932025-05-30T21:33:15.944ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.099747ms iteration = 15 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
25942025-05-30T21:33:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25952025-05-30T21:33:15.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25962025-05-30T21:33:15.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25972025-05-30T21:33:15.947ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25982025-05-30T21:33:15.948ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25992025-05-30T21:33:15.948ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26002025-05-30T21:33:15.949ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
26012025-05-30T21:33:15.949ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26022025-05-30T21:33:15.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26032025-05-30T21:33:15.950ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
26042025-05-30T21:33:15.950ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26052025-05-30T21:33:15.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26062025-05-30T21:33:15.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
26072025-05-30T21:33:15.951ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26082025-05-30T21:33:15.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26092025-05-30T21:33:15.951ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26102025-05-30T21:33:15.952ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-30 21:23:15.940350943 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
26112025-05-30T21:33:15.953ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 12.627865ms iteration = 8 status = {"expiration":"2025-05-30T21:23:15.940350943Z","pruned":[]}
26122025-05-30T21:33:15.953ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
26132025-05-30T21:33:15.953ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
26142025-05-30T21:33:15.953ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:202 num_receivers = 0
26152025-05-30T21:33:15.953ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 12.855067ms iteration = 8 status = {"by_rx":{},"error":null}
26162025-05-30T21:33:15.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
26172025-05-30T21:33:15.953ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26182025-05-30T21:33:15.955ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1530, tv_nsec: 892914293 } })
26192025-05-30T21:33:15.955ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
26202025-05-30T21:33:15.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
26212025-05-30T21:33:15.955ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
26222025-05-30T21:33:15.955ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
26232025-05-30T21:33:15.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
26242025-05-30T21:33:15.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
26252025-05-30T21:33:15.956ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 15.68104ms iteration = 8 status = {"errors":[],"found_port_ok":[]}
26262025-05-30T21:33:15.958ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 17.607052ms iteration = 15 status = {"errors":[],"garbage_collect_requested":[]}
26272025-05-30T21:33:15.959ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26282025-05-30T21:33:15.959ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26292025-05-30T21:33:15.959ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1530, tv_nsec: 897429114 } })
26302025-05-30T21:33:15.960ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
26312025-05-30T21:33:15.960ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26322025-05-30T21:33:15.962ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
26332025-05-30T21:33:15.962ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
26342025-05-30T21:33:15.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
26352025-05-30T21:33:15.962ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
26362025-05-30T21:33:15.962ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
26372025-05-30T21:33:15.962ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
26382025-05-30T21:33:15.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
26392025-05-30T21:33:15.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
26402025-05-30T21:33:15.964ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
26412025-05-30T21:33:15.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26422025-05-30T21:33:15.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26432025-05-30T21:33:15.966ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
26442025-05-30T21:33:15.966ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
26452025-05-30T21:33:15.966ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
26462025-05-30T21:33:15.966ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
26472025-05-30T21:33:15.966ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26482025-05-30T21:33:15.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
26492025-05-30T21:33:15.967ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26502025-05-30T21:33:15.968ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26512025-05-30T21:33:15.968ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1530, tv_nsec: 905876551 } })
26522025-05-30T21:33:15.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26532025-05-30T21:33:15.968ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
26542025-05-30T21:33:15.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
26552025-05-30T21:33:15.968ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
26562025-05-30T21:33:15.969ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
26572025-05-30T21:33:15.969ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26582025-05-30T21:33:15.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26592025-05-30T21:33:15.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
26602025-05-30T21:33:15.975ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26612025-05-30T21:33:15.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26622025-05-30T21:33:15.976ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26632025-05-30T21:33:15.976ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 35.850882ms iteration = 8 status = {"dispatched":[],"errors":[],"glob_version":"144.0.0","globs_reprocessed":{},"no_receivers":[]}
26642025-05-30T21:33:15.976ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
26652025-05-30T21:33:15.976ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
26662025-05-30T21:33:15.976ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 35.63853ms iteration = 8 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
26672025-05-30T21:33:15.978ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26682025-05-30T21:33:15.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
26692025-05-30T21:33:15.979ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26702025-05-30T21:33:15.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26712025-05-30T21:33:15.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
26722025-05-30T21:33:15.979ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26732025-05-30T21:33:15.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26742025-05-30T21:33:15.980ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
26752025-05-30T21:33:15.981ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
26762025-05-30T21:33:15.981ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
26772025-05-30T21:33:15.981ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
26782025-05-30T21:33:15.981ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 40.91566ms iteration = 8 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
26792025-05-30T21:33:15.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
26802025-05-30T21:33:15.983ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
26812025-05-30T21:33:15.983ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
26822025-05-30T21:33:15.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
26832025-05-30T21:33:15.989ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 49.662141ms iteration = 15 status = {"errors":[],"finish_invoked_ok":[]}
26842025-05-30T21:33:15.991ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26852025-05-30T21:33:15.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26862025-05-30T21:33:15.997ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
26872025-05-30T21:33:15.997ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
26882025-05-30T21:33:15.997ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
26892025-05-30T21:33:15.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
26902025-05-30T21:33:15.998ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
26912025-05-30T21:33:15.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
26922025-05-30T21:33:16.002ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
26932025-05-30T21:33:16.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
26942025-05-30T21:33:16.003ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
26952025-05-30T21:33:16.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
26962025-05-30T21:33:16.009ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26972025-05-30T21:33:16.012ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
26982025-05-30T21:33:16.012ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26992025-05-30T21:33:16.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27002025-05-30T21:33:16.014ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 73.567154ms iteration = 15 status = {"errors":[],"requests_created_ok":[]}
27012025-05-30T21:33:16.023ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
27022025-05-30T21:33:16.023ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
27032025-05-30T21:33:16.023ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
27042025-05-30T21:33:16.023ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
27052025-05-30T21:33:16.023ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27062025-05-30T21:33:16.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
27072025-05-30T21:33:16.023ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
27082025-05-30T21:33:16.023ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 82.957284ms iteration = 15 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
27092025-05-30T21:33:16.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27102025-05-30T21:33:16.028ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
27112025-05-30T21:33:16.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
27122025-05-30T21:33:16.079ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
27132025-05-30T21:33:16.087ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
27142025-05-30T21:33:16.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
27152025-05-30T21:33:16.101ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
27162025-05-30T21:33:16.162ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
27172025-05-30T21:33:16.162ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
27182025-05-30T21:33:16.163ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e778a827-4c9c-4899-945d-bf76dff3613c", "content-length": "26", "date": "Fri, 30 May 2025 21:33:16 GMT"} })
27192025-05-30T21:33:16.163ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
27202025-05-30T21:33:16.165ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
27212025-05-30T21:33:16.186ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
27222025-05-30T21:33:16.186ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
27232025-05-30T21:33:16.187ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "06b1c947-9a3d-4959-b665-ed7602ba813f", "content-length": "372", "date": "Fri, 30 May 2025 21:33:15 GMT"} })
27242025-05-30T21:33:16.187ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27252025-05-30T21:33:16.187ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27262025-05-30T21:33:16.191ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 250.71184ms iteration = 43 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
27272025-05-30T21:33:16.193ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27282025-05-30T21:33:16.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27292025-05-30T21:33:16.202ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
27302025-05-30T21:33:16.202ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 262.581156ms iteration = 43 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
27312025-05-30T21:33:16.228ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
27322025-05-30T21:33:16.228ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
27332025-05-30T21:33:16.229ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "be1f2a94-ac70-4e0c-b347-35ee26ee6878", "date": "Fri, 30 May 2025 21:33:15 GMT"} })
27342025-05-30T21:33:16.229ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27352025-05-30T21:33:16.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27362025-05-30T21:33:16.231ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27372025-05-30T21:33:16.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27382025-05-30T21:33:16.235ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27392025-05-30T21:33:16.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27402025-05-30T21:33:16.238ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 298.064713ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
27412025-05-30T21:33:16.239ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
27422025-05-30T21:33:16.239ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
27432025-05-30T21:33:16.240ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "93f1f3f8-7123-4527-89d3-29486787a46d", "date": "Fri, 30 May 2025 21:33:15 GMT"} })
27442025-05-30T21:33:16.241ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27452025-05-30T21:33:16.241ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27462025-05-30T21:33:16.241ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27472025-05-30T21:33:16.242ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 302.490884ms iteration = 15 status = {}
27482025-05-30T21:33:16.242ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27492025-05-30T21:33:16.244ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
27502025-05-30T21:33:16.244ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
27512025-05-30T21:33:16.244ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
27522025-05-30T21:33:16.244ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
27532025-05-30T21:33:16.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
27542025-05-30T21:33:16.244ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
27552025-05-30T21:33:16.244ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
27562025-05-30T21:33:16.245ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
27572025-05-30T21:33:16.245ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
27582025-05-30T21:33:16.245ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
27592025-05-30T21:33:16.245ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
27602025-05-30T21:33:16.246ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
27612025-05-30T21:33:16.246ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27622025-05-30T21:33:16.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27632025-05-30T21:33:16.246ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 306.409589ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
27642025-05-30T21:33:16.246ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
27652025-05-30T21:33:16.246ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
27662025-05-30T21:33:16.247ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7ba8e6b-46fe-4a03-bc74-e3fa96b1ed11", "content-length": "26", "date": "Fri, 30 May 2025 21:33:16 GMT"} })
27672025-05-30T21:33:16.247ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
27682025-05-30T21:33:16.247ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27692025-05-30T21:33:16.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27702025-05-30T21:33:16.248ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27712025-05-30T21:33:16.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27722025-05-30T21:33:16.249ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
27732025-05-30T21:33:16.259ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
27742025-05-30T21:33:16.259ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
27752025-05-30T21:33:16.262ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
27762025-05-30T21:33:16.262ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
27772025-05-30T21:33:16.262ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac942d9e-76a5-4915-8f8e-e9e44c12c763", "content-length": "26", "date": "Fri, 30 May 2025 21:33:16 GMT"} })
27782025-05-30T21:33:16.262ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
27792025-05-30T21:33:16.263ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
27802025-05-30T21:33:16.265ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
27812025-05-30T21:33:16.287ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27822025-05-30T21:33:16.289ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
27832025-05-30T21:33:16.290ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "9bc3e239-201f-469e-9083-00f99e558f27", "date": "Fri, 30 May 2025 21:33:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
27842025-05-30T21:33:16.290ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
27852025-05-30T21:33:16.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
27862025-05-30T21:33:16.291ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
27872025-05-30T21:33:16.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
27882025-05-30T21:33:16.291ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "833ea1a3-cb54-4d75-abda-34157965e301", "date": "Fri, 30 May 2025 21:33:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
27892025-05-30T21:33:16.291ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
27902025-05-30T21:33:16.294ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
27912025-05-30T21:33:16.294ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 353.377268ms iteration = 10 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
27922025-05-30T21:33:16.301ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
27932025-05-30T21:33:16.301ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 0 version = 2
27942025-05-30T21:33:16.301ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
27952025-05-30T21:33:16.301ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
27962025-05-30T21:33:16.301ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
27972025-05-30T21:33:16.301ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-30 21:26:17.226115 UTC dns_group = internal generation = 1
27982025-05-30T21:33:16.301ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 362.042178ms iteration = 8 status = {"generation":1}
27992025-05-30T21:33:16.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28002025-05-30T21:33:16.303ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28012025-05-30T21:33:16.304ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 363.565716ms iteration = 15 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
28022025-05-30T21:33:16.304ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
28032025-05-30T21:33:16.304ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
28042025-05-30T21:33:16.304ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 363.712887ms iteration = 15 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
28052025-05-30T21:33:16.305ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
28062025-05-30T21:33:16.305ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28072025-05-30T21:33:16.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28082025-05-30T21:33:16.308ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28092025-05-30T21:33:16.311ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
28102025-05-30T21:33:16.311ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28112025-05-30T21:33:16.311ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
28122025-05-30T21:33:16.311ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 370.840859ms iteration = 15 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
28132025-05-30T21:33:16.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28142025-05-30T21:33:16.314ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 373.5297ms iteration = 15 status = 0
28152025-05-30T21:33:16.335ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28162025-05-30T21:33:16.340ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
28172025-05-30T21:33:16.340ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28182025-05-30T21:33:16.340ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28192025-05-30T21:33:16.342ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 1 version = 2
28202025-05-30T21:33:16.342ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
28212025-05-30T21:33:16.342ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-30 21:26:18.957109 UTC dns_group = external generation = 2
28222025-05-30T21:33:16.342ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 401.900986ms iteration = 8 status = {"generation":2}
28232025-05-30T21:33:16.344ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 403.891899ms iteration = 16 status = {"physical_disks_added":0}
28242025-05-30T21:33:16.344ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28252025-05-30T21:33:16.345ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28262025-05-30T21:33:16.346ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28272025-05-30T21:33:16.346ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-30T21:26:13.986396Z, time_modified: 2025-05-30T21:26:13.986396Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }) }
28282025-05-30T21:33:16.346ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 406.5531ms iteration = 8 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]}},"default_endpoint":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
28292025-05-30T21:33:16.348ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
28302025-05-30T21:33:16.348ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28312025-05-30T21:33:16.348ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28322025-05-30T21:33:16.364ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
28332025-05-30T21:33:16.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
28342025-05-30T21:33:16.367ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 426.580439ms iteration = 15 status = null
28352025-05-30T21:33:16.381ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
28362025-05-30T21:33:16.382ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "eca7e3a5-5cc5-4be3-acde-eaf082454aa8", "date": "Fri, 30 May 2025 21:33:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
28372025-05-30T21:33:16.382ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
28382025-05-30T21:33:16.382ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 442.017667ms iteration = 15 status = {}
28392025-05-30T21:33:16.384ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "46f4a1ad-804c-4ccd-90f9-ceb155a86cc8", "date": "Fri, 30 May 2025 21:33:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
28402025-05-30T21:33:16.384ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
28412025-05-30T21:33:16.386ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "2be4d225-c110-4563-87c1-0eaa0f43cdb2", "date": "Fri, 30 May 2025 21:33:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
28422025-05-30T21:33:16.386ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
28432025-05-30T21:33:16.388ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28442025-05-30T21:33:16.388ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
28452025-05-30T21:33:16.388ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 448.3483ms iteration = 10 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
28462025-05-30T21:33:16.390ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
28472025-05-30T21:33:16.390ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 449.736365ms iteration = 8 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
28482025-05-30T21:33:16.390ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
28492025-05-30T21:33:16.390ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28502025-05-30T21:33:16.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28512025-05-30T21:33:16.425ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28522025-05-30T21:33:16.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28532025-05-30T21:33:16.427ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
28542025-05-30T21:33:16.427ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28552025-05-30T21:33:16.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28562025-05-30T21:33:16.439ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
28572025-05-30T21:33:16.439ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
28582025-05-30T21:33:16.439ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
28592025-05-30T21:33:16.439ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
28602025-05-30T21:33:16.439ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
28612025-05-30T21:33:16.439ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
28622025-05-30T21:33:16.439ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
28632025-05-30T21:33:16.439ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
28642025-05-30T21:33:16.439ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
28652025-05-30T21:33:16.439ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
28662025-05-30T21:33:16.439ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
28672025-05-30T21:33:16.439ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28682025-05-30T21:33:16.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28692025-05-30T21:33:16.442ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
28702025-05-30T21:33:16.442ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
28712025-05-30T21:33:16.487ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
28722025-05-30T21:33:16.550ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "6d9f8260-63fa-4e5a-a976-6413c380fd56", "date": "Fri, 30 May 2025 21:33:15 GMT"} })
28732025-05-30T21:33:16.550ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
28742025-05-30T21:33:16.550ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
28752025-05-30T21:33:16.550ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "efeaf137-4da2-4a74-b282-031399856869", "content-length": "682", "date": "Fri, 30 May 2025 21:33:15 GMT"} })
28762025-05-30T21:33:16.550ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28772025-05-30T21:33:16.551ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28782025-05-30T21:33:16.551ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdf00eb5-1eee-4f76-8107-5279ff219273", "content-length": "691", "date": "Fri, 30 May 2025 21:33:16 GMT"} })
28792025-05-30T21:33:16.551ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
28802025-05-30T21:33:16.551ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
28812025-05-30T21:33:16.551ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
28822025-05-30T21:33:16.552ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 612.0947ms iteration = 15 status = {}
28832025-05-30T21:33:16.555ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28842025-05-30T21:33:16.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28852025-05-30T21:33:16.559ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28862025-05-30T21:33:16.559ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28872025-05-30T21:33:16.562ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28882025-05-30T21:33:16.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28892025-05-30T21:33:16.568ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 628.423358ms iteration = 15 status = {}
28902025-05-30T21:33:18.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 367 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 25a82b63-07e9-405a-9e75-a1b85caecdbe response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
28912025-05-30T21:33:25.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
28922025-05-30T21:33:25.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 44 reason = Timeout
28932025-05-30T21:33:25.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
28942025-05-30T21:33:25.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28952025-05-30T21:33:25.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28962025-05-30T21:33:25.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
28972025-05-30T21:33:25.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28982025-05-30T21:33:25.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
28992025-05-30T21:33:25.979ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29002025-05-30T21:33:25.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29012025-05-30T21:33:25.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29022025-05-30T21:33:25.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29032025-05-30T21:33:25.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29042025-05-30T21:33:25.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29052025-05-30T21:33:25.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29062025-05-30T21:33:25.986ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
29072025-05-30T21:33:25.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29082025-05-30T21:33:25.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29092025-05-30T21:33:25.998ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
29102025-05-30T21:33:25.998ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 57.745823ms iteration = 44 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
29112025-05-30T21:33:26.017ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 76.688581ms iteration = 44 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
29122025-05-30T21:33:28.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 310 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = d224e844-078b-4af2-81e5-805057eac7b9 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
29132025-05-30T21:33:35.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 45 reason = Timeout
29142025-05-30T21:33:35.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29152025-05-30T21:33:35.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
29162025-05-30T21:33:35.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29172025-05-30T21:33:35.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29182025-05-30T21:33:35.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29192025-05-30T21:33:35.971ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29202025-05-30T21:33:35.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29212025-05-30T21:33:35.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29222025-05-30T21:33:35.977ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
29232025-05-30T21:33:35.977ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29242025-05-30T21:33:35.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29252025-05-30T21:33:35.980ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29262025-05-30T21:33:35.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29272025-05-30T21:33:35.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29282025-05-30T21:33:35.984ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
29292025-05-30T21:33:35.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29302025-05-30T21:33:35.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29312025-05-30T21:33:35.997ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
29322025-05-30T21:33:35.997ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 57.007705ms iteration = 45 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
29332025-05-30T21:33:36.008ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 67.86112ms iteration = 45 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
29342025-05-30T21:33:38.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 301 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 5e2583bb-0211-4bd9-8f8f-ea4e9266d7c6 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
29352025-05-30T21:33:45.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 46 reason = Timeout
29362025-05-30T21:33:45.940ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 16 reason = Timeout
29372025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29382025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
29392025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
29402025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
29412025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 16 reason = Timeout
29422025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
29432025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29442025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 17 reason = Timeout
29452025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
29462025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 16 reason = Timeout
29472025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
29482025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
29492025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 16 reason = Timeout
29502025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
29512025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
29522025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
29532025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 16 reason = Timeout
29542025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 16 reason = Timeout
29552025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
29562025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
29572025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 16 reason = Timeout
29582025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29592025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 16 reason = Timeout
29602025-05-30T21:33:45.941ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
29612025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
29622025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29632025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
29642025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 16 reason = Timeout
29652025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
29662025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
29672025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 16 reason = Timeout
29682025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
29692025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29702025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 16 reason = Timeout
29712025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
29722025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
29732025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29742025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29752025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
29762025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29772025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
29782025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
29792025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
29802025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
29812025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
29822025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
29832025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
29842025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29852025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
29862025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29872025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
29882025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29892025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
29902025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
29912025-05-30T21:33:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29922025-05-30T21:33:45.944ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.100616ms iteration = 16 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
29932025-05-30T21:33:45.944ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29942025-05-30T21:33:45.944ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1560, tv_nsec: 882349475 } })
29952025-05-30T21:33:45.945ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
29962025-05-30T21:33:45.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
29972025-05-30T21:33:45.947ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29982025-05-30T21:33:45.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29992025-05-30T21:33:45.952ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
30002025-05-30T21:33:45.952ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
30012025-05-30T21:33:45.952ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
30022025-05-30T21:33:45.952ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30032025-05-30T21:33:45.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
30042025-05-30T21:33:45.954ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1560, tv_nsec: 891647712 } })
30052025-05-30T21:33:45.954ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
30062025-05-30T21:33:45.956ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
30072025-05-30T21:33:45.956ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30082025-05-30T21:33:45.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30092025-05-30T21:33:45.958ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30102025-05-30T21:33:45.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30112025-05-30T21:33:45.959ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
30122025-05-30T21:33:45.959ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
30132025-05-30T21:33:45.959ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
30142025-05-30T21:33:45.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
30152025-05-30T21:33:45.960ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1560, tv_nsec: 898018255 } })
30162025-05-30T21:33:45.960ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
30172025-05-30T21:33:45.979ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
30182025-05-30T21:33:45.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
30192025-05-30T21:33:45.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
30202025-05-30T21:33:45.984ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 42.993394ms iteration = 16 status = {"errors":[],"finish_invoked_ok":[]}
30212025-05-30T21:33:45.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
30222025-05-30T21:33:45.984ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 43.771052ms iteration = 16 status = {"errors":[],"requests_created_ok":[]}
30232025-05-30T21:33:45.985ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30242025-05-30T21:33:45.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30252025-05-30T21:33:45.986ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30262025-05-30T21:33:45.986ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 45.632554ms iteration = 16 status = {"errors":[],"garbage_collect_requested":[]}
30272025-05-30T21:33:45.987ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30282025-05-30T21:33:45.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30292025-05-30T21:33:45.988ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
30302025-05-30T21:33:45.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
30312025-05-30T21:33:45.989ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
30322025-05-30T21:33:45.989ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30332025-05-30T21:33:45.989ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30342025-05-30T21:33:45.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30352025-05-30T21:33:45.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30362025-05-30T21:33:45.990ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
30372025-05-30T21:33:45.990ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30382025-05-30T21:33:45.990ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
30392025-05-30T21:33:45.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30402025-05-30T21:33:45.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30412025-05-30T21:33:45.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30422025-05-30T21:33:45.991ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
30432025-05-30T21:33:45.991ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30442025-05-30T21:33:45.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30452025-05-30T21:33:45.992ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30462025-05-30T21:33:45.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30472025-05-30T21:33:45.992ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30482025-05-30T21:33:45.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30492025-05-30T21:33:45.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30502025-05-30T21:33:45.993ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 52.355501ms iteration = 46 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
30512025-05-30T21:33:45.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
30522025-05-30T21:33:45.994ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
30532025-05-30T21:33:45.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
30542025-05-30T21:33:45.996ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
30552025-05-30T21:33:45.996ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30562025-05-30T21:33:45.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30572025-05-30T21:33:46.005ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
30582025-05-30T21:33:46.005ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
30592025-05-30T21:33:46.005ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 64.214167ms iteration = 16 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
30602025-05-30T21:33:46.005ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30612025-05-30T21:33:46.007ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
30622025-05-30T21:33:46.008ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
30632025-05-30T21:33:46.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
30642025-05-30T21:33:46.039ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30652025-05-30T21:33:46.040ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
30662025-05-30T21:33:46.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
30672025-05-30T21:33:46.042ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30682025-05-30T21:33:46.042ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
30692025-05-30T21:33:46.042ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30702025-05-30T21:33:46.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30712025-05-30T21:33:46.044ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
30722025-05-30T21:33:46.044ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30732025-05-30T21:33:46.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30742025-05-30T21:33:46.045ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 104.693892ms iteration = 16 status = 0
30752025-05-30T21:33:46.046ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 104.867924ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
30762025-05-30T21:33:46.046ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 105.682104ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
30772025-05-30T21:33:46.046ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
30782025-05-30T21:33:46.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
30792025-05-30T21:33:46.048ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30802025-05-30T21:33:46.050ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30812025-05-30T21:33:46.050ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
30822025-05-30T21:33:46.050ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
30832025-05-30T21:33:46.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
30842025-05-30T21:33:46.053ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
30852025-05-30T21:33:46.053ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30862025-05-30T21:33:46.053ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30872025-05-30T21:33:46.064ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
30882025-05-30T21:33:46.064ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
30892025-05-30T21:33:46.064ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b54d145e-635f-4bf4-9dc5-bf3afdbefbfa", "content-length": "26", "date": "Fri, 30 May 2025 21:33:46 GMT"} })
30902025-05-30T21:33:46.064ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
30912025-05-30T21:33:46.066ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
30922025-05-30T21:33:46.067ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
30932025-05-30T21:33:46.067ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 126.588734ms iteration = 16 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
30942025-05-30T21:33:46.068ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
30952025-05-30T21:33:46.068ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 127.697817ms iteration = 46 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
30962025-05-30T21:33:46.086ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 145.851094ms iteration = 16 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
30972025-05-30T21:33:46.090ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
30982025-05-30T21:33:46.091ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
30992025-05-30T21:33:46.091ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3039bb1f-5500-445c-9e93-6aed67dc7282", "content-length": "26", "date": "Fri, 30 May 2025 21:33:46 GMT"} })
31002025-05-30T21:33:46.091ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
31012025-05-30T21:33:46.093ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
31022025-05-30T21:33:46.110ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
31032025-05-30T21:33:46.110ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 169.558927ms iteration = 16 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
31042025-05-30T21:33:46.112ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 171.700981ms iteration = 17 status = {"physical_disks_added":0}
31052025-05-30T21:33:46.113ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
31062025-05-30T21:33:46.113ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
31072025-05-30T21:33:46.113ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "09cdcd40-1ea1-46a9-a040-f61fe55f25f3", "content-length": "26", "date": "Fri, 30 May 2025 21:33:46 GMT"} })
31082025-05-30T21:33:46.113ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
31092025-05-30T21:33:46.115ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
31102025-05-30T21:33:46.117ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31112025-05-30T21:33:46.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31122025-05-30T21:33:46.189ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
31132025-05-30T21:33:46.189ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
31142025-05-30T21:33:46.192ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 251.181894ms iteration = 16 status = null
31152025-05-30T21:33:46.196ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
31162025-05-30T21:33:46.196ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
31172025-05-30T21:33:46.196ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "4826b801-c333-440c-aa59-b823ec8984ac", "content-length": "372", "date": "Fri, 30 May 2025 21:33:45 GMT"} })
31182025-05-30T21:33:46.196ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31192025-05-30T21:33:46.197ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31202025-05-30T21:33:46.197ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 256.542236ms iteration = 16 status = {}
31212025-05-30T21:33:46.224ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31222025-05-30T21:33:46.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31232025-05-30T21:33:46.248ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
31242025-05-30T21:33:46.248ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
31252025-05-30T21:33:46.248ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31262025-05-30T21:33:46.248ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "06965b39-706f-4232-941d-cf58ffe651df", "date": "Fri, 30 May 2025 21:33:45 GMT"} })
31272025-05-30T21:33:46.248ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31282025-05-30T21:33:46.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31292025-05-30T21:33:46.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31302025-05-30T21:33:46.251ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
31312025-05-30T21:33:46.251ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31322025-05-30T21:33:46.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31332025-05-30T21:33:46.252ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31342025-05-30T21:33:46.252ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31352025-05-30T21:33:46.255ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31362025-05-30T21:33:46.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31372025-05-30T21:33:46.255ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31382025-05-30T21:33:46.257ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
31392025-05-30T21:33:46.257ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31402025-05-30T21:33:46.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31412025-05-30T21:33:46.258ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
31422025-05-30T21:33:46.258ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
31432025-05-30T21:33:46.258ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "d6609d92-2338-45ca-9ca3-a28e5faa32c6", "date": "Fri, 30 May 2025 21:33:45 GMT"} })
31442025-05-30T21:33:46.259ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31452025-05-30T21:33:46.260ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 319.064304ms iteration = 16 status = {}
31462025-05-30T21:33:46.261ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
31472025-05-30T21:33:46.261ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31482025-05-30T21:33:46.261ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31492025-05-30T21:33:46.263ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
31502025-05-30T21:33:46.263ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
31512025-05-30T21:33:46.264ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
31522025-05-30T21:33:46.264ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
31532025-05-30T21:33:46.264ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
31542025-05-30T21:33:46.264ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
31552025-05-30T21:33:46.264ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
31562025-05-30T21:33:46.264ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
31572025-05-30T21:33:46.264ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
31582025-05-30T21:33:46.264ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
31592025-05-30T21:33:46.264ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
31602025-05-30T21:33:46.264ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31612025-05-30T21:33:46.264ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31622025-05-30T21:33:46.266ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
31632025-05-30T21:33:46.266ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
31642025-05-30T21:33:46.355ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
31652025-05-30T21:33:46.374ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "3d61b902-d082-457a-8c89-eca093490b6c", "date": "Fri, 30 May 2025 21:33:45 GMT"} })
31662025-05-30T21:33:46.374ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
31672025-05-30T21:33:46.374ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
31682025-05-30T21:33:46.375ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "865c90ce-f8b8-4735-9948-88524d82768e", "content-length": "682", "date": "Fri, 30 May 2025 21:33:46 GMT"} })
31692025-05-30T21:33:46.375ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31702025-05-30T21:33:46.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31712025-05-30T21:33:46.378ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31722025-05-30T21:33:46.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31732025-05-30T21:33:46.379ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "042f711e-4fe0-4cf4-b836-4276512f8eeb", "content-length": "691", "date": "Fri, 30 May 2025 21:33:46 GMT"} })
31742025-05-30T21:33:46.379ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
31752025-05-30T21:33:46.379ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
31762025-05-30T21:33:46.379ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
31772025-05-30T21:33:46.380ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 439.601079ms iteration = 16 status = {}
31782025-05-30T21:33:46.381ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31792025-05-30T21:33:46.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31802025-05-30T21:33:46.384ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31812025-05-30T21:33:46.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31822025-05-30T21:33:46.391ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 449.891347ms iteration = 16 status = {}
31832025-05-30T21:33:48.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 316 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = afcf64e0-fb8e-461c-865d-cf9e7979143a response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
31842025-05-30T21:33:55.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
31852025-05-30T21:33:55.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 47 reason = Timeout
31862025-05-30T21:33:55.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31872025-05-30T21:33:55.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31882025-05-30T21:33:55.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31892025-05-30T21:33:55.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31902025-05-30T21:33:55.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31912025-05-30T21:33:55.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31922025-05-30T21:33:55.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31932025-05-30T21:33:55.943ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.092526ms iteration = 47 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
31942025-05-30T21:33:55.945ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
31952025-05-30T21:33:55.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31962025-05-30T21:33:55.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31972025-05-30T21:33:55.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31982025-05-30T21:33:55.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31992025-05-30T21:33:55.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32002025-05-30T21:33:55.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
32012025-05-30T21:33:55.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32022025-05-30T21:33:55.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32032025-05-30T21:33:55.963ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
32042025-05-30T21:33:55.964ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.456749ms iteration = 47 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
32052025-05-30T21:33:58.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 316 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = e8f701f2-2837-40f2-b89d-93b1841a63c8 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
32062025-05-30T21:34:05.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
32072025-05-30T21:34:05.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 48 reason = Timeout
32082025-05-30T21:34:05.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32092025-05-30T21:34:05.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32102025-05-30T21:34:05.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32112025-05-30T21:34:05.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32122025-05-30T21:34:05.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32132025-05-30T21:34:05.945ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
32142025-05-30T21:34:05.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32152025-05-30T21:34:05.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32162025-05-30T21:34:05.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32172025-05-30T21:34:05.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32182025-05-30T21:34:05.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32192025-05-30T21:34:05.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
32202025-05-30T21:34:05.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32212025-05-30T21:34:05.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32222025-05-30T21:34:05.960ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
32232025-05-30T21:34:05.960ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.420833ms iteration = 48 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
32242025-05-30T21:34:05.973ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32252025-05-30T21:34:05.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32262025-05-30T21:34:06.004ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 63.784103ms iteration = 48 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
32272025-05-30T21:34:08.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 306 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = d672764a-f5e7-4dfb-95ee-0f23503c0e15 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
32282025-05-30T21:34:13.765ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:64220
32292025-05-30T21:34:13.765ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 84e71c3b-f1e3-4867-8f98-660713dbe6c6 roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
32302025-05-30T21:34:13.766ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 84e71c3b-f1e3-4867-8f98-660713dbe6c6 resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
32312025-05-30T21:34:13.769ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3417 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 84e71c3b-f1e3-4867-8f98-660713dbe6c6 response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
32322025-05-30T21:34:13.769ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = bd88dc3e-c3c5-47d9-b432-e485db479f00 roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
32332025-05-30T21:34:13.769ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = bd88dc3e-c3c5-47d9-b432-e485db479f00 resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
32342025-05-30T21:34:13.772ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2647 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = bd88dc3e-c3c5-47d9-b432-e485db479f00 response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
32352025-05-30T21:34:15.939ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
32362025-05-30T21:34:15.939ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
32372025-05-30T21:34:15.939ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
32382025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
32392025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
32402025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 17 reason = Timeout
32412025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 17 reason = Timeout
32422025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
32432025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 17 reason = Timeout
32442025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
32452025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
32462025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 17 reason = Timeout
32472025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
32482025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
32492025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
32502025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32512025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 17 reason = Timeout
32522025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
32532025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 17 reason = Timeout
32542025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
32552025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
32562025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
32572025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 17 reason = Timeout
32582025-05-30T21:34:15.940ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
32592025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32602025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
32612025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
32622025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 18 reason = Timeout
32632025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
32642025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
32652025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
32662025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 17 reason = Timeout
32672025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32682025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 17 reason = Timeout
32692025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32702025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
32712025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
32722025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 49 reason = Timeout
32732025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32742025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 9 reason = Timeout
32752025-05-30T21:34:15.940ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
32762025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
32772025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32782025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 9 reason = Timeout
32792025-05-30T21:34:15.940ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-30 21:24:15.940853198 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
32802025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
32812025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
32822025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
32832025-05-30T21:34:15.940ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e
32842025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.76\u{b5}s iteration = 10 status = {"enabled":false,"target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e"}
32852025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
32862025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 9 reason = Timeout
32872025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 9 reason = Timeout
32882025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
32892025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
32902025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
32912025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
32922025-05-30T21:34:15.940ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
32932025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
32942025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
32952025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
32962025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 9 reason = Timeout
32972025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 9 reason = Timeout
32982025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
32992025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
33002025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 9 reason = Timeout
33012025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
33022025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
33032025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 17 reason = Timeout
33042025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
33052025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 17 reason = Timeout
33062025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
33072025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
33082025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
33092025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
33102025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33112025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
33122025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
33132025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
33142025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
33152025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
33162025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
33172025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33182025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
33192025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33202025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
33212025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33222025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33232025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
33242025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33252025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
33262025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
33272025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
33282025-05-30T21:34:15.941ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2a161087-f464-41f8-b020-0f9e83fd5351.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1590, tv_nsec: 878887735 } })
33292025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
33302025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33312025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
33322025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
33332025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
33342025-05-30T21:34:15.941ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0413f6b4-86a1-42a1-919a-e8f42ff9f638.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("acb1452d-0a02-4b36-a462-c8ce2f96a488.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:2::1))) }], valid_until: Instant { tv_sec: 1590, tv_nsec: 878939426 } })
33352025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
33362025-05-30T21:34:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
33372025-05-30T21:34:15.942ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
33382025-05-30T21:34:15.942ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.142933ms iteration = 9 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
33392025-05-30T21:34:15.942ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } dns_group = internal
33402025-05-30T21:34:15.942ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.175884ms iteration = 9 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353"]}
33412025-05-30T21:34:15.942ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33422025-05-30T21:34:15.944ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.607932ms iteration = 17 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
33432025-05-30T21:34:15.945ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
33442025-05-30T21:34:15.945ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
33452025-05-30T21:34:15.945ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
33462025-05-30T21:34:15.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
33472025-05-30T21:34:15.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
33482025-05-30T21:34:15.948ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33492025-05-30T21:34:15.948ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33502025-05-30T21:34:15.950ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
33512025-05-30T21:34:15.950ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
33522025-05-30T21:34:15.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
33532025-05-30T21:34:15.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
33542025-05-30T21:34:15.951ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
33552025-05-30T21:34:15.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
33562025-05-30T21:34:15.952ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33572025-05-30T21:34:15.953ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1590, tv_nsec: 890854443 } })
33582025-05-30T21:34:15.953ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
33592025-05-30T21:34:15.954ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
33602025-05-30T21:34:15.954ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33612025-05-30T21:34:15.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33622025-05-30T21:34:15.962ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33632025-05-30T21:34:15.963ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1590, tv_nsec: 901061520 } })
33642025-05-30T21:34:15.964ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
33652025-05-30T21:34:15.964ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
33662025-05-30T21:34:15.964ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33672025-05-30T21:34:15.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33682025-05-30T21:34:15.967ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-30 21:24:15.940853198 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
33692025-05-30T21:34:15.967ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 26.726376ms iteration = 9 status = {"expiration":"2025-05-30T21:24:15.940853198Z","pruned":[]}
33702025-05-30T21:34:15.969ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 28.205964ms iteration = 9 status = {"errors":[],"found_port_ok":[]}
33712025-05-30T21:34:15.969ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
33722025-05-30T21:34:15.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
33732025-05-30T21:34:15.971ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:202 num_receivers = 0
33742025-05-30T21:34:15.971ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 30.025965ms iteration = 9 status = {"by_rx":{},"error":null}
33752025-05-30T21:34:15.993ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 53.10164ms iteration = 17 status = {"errors":[],"requests_created_ok":[]}
33762025-05-30T21:34:15.995ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
33772025-05-30T21:34:15.995ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
33782025-05-30T21:34:15.995ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
33792025-05-30T21:34:15.995ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
33802025-05-30T21:34:15.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
33812025-05-30T21:34:15.998ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33822025-05-30T21:34:15.998ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33832025-05-30T21:34:15.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33842025-05-30T21:34:16.002ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33852025-05-30T21:34:16.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
33862025-05-30T21:34:16.003ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
33872025-05-30T21:34:16.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33882025-05-30T21:34:16.005ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33892025-05-30T21:34:16.006ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
33902025-05-30T21:34:16.006ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
33912025-05-30T21:34:16.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
33922025-05-30T21:34:16.006ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33932025-05-30T21:34:16.007ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
33942025-05-30T21:34:16.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
33952025-05-30T21:34:16.008ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
33962025-05-30T21:34:16.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
33972025-05-30T21:34:16.009ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
33982025-05-30T21:34:16.009ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33992025-05-30T21:34:16.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34002025-05-30T21:34:16.009ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
34012025-05-30T21:34:16.009ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 68.675629ms iteration = 17 status = {"errors":[],"finish_invoked_ok":[]}
34022025-05-30T21:34:16.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
34032025-05-30T21:34:16.010ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
34042025-05-30T21:34:16.010ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
34052025-05-30T21:34:16.010ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
34062025-05-30T21:34:16.010ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
34072025-05-30T21:34:16.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
34082025-05-30T21:34:16.011ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34092025-05-30T21:34:16.011ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34102025-05-30T21:34:16.012ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34112025-05-30T21:34:16.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34122025-05-30T21:34:16.013ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34132025-05-30T21:34:16.013ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 72.655925ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
34142025-05-30T21:34:16.013ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 72.912288ms iteration = 9 status = {"dispatched":[],"errors":[],"glob_version":"144.0.0","globs_reprocessed":{},"no_receivers":[]}
34152025-05-30T21:34:16.014ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
34162025-05-30T21:34:16.014ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
34172025-05-30T21:34:16.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
34182025-05-30T21:34:16.015ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
34192025-05-30T21:34:16.015ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34202025-05-30T21:34:16.015ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
34212025-05-30T21:34:16.015ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
34222025-05-30T21:34:16.015ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34232025-05-30T21:34:16.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
34242025-05-30T21:34:16.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
34252025-05-30T21:34:16.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34262025-05-30T21:34:16.018ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
34272025-05-30T21:34:16.018ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
34282025-05-30T21:34:16.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
34292025-05-30T21:34:16.018ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
34302025-05-30T21:34:16.018ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34312025-05-30T21:34:16.018ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
34322025-05-30T21:34:16.018ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34332025-05-30T21:34:16.018ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34342025-05-30T21:34:16.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
34352025-05-30T21:34:16.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34362025-05-30T21:34:16.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34372025-05-30T21:34:16.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
34382025-05-30T21:34:16.021ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34392025-05-30T21:34:16.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34402025-05-30T21:34:16.024ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 84.218557ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
34412025-05-30T21:34:16.025ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
34422025-05-30T21:34:16.026ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
34432025-05-30T21:34:16.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
34442025-05-30T21:34:16.027ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34452025-05-30T21:34:16.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34462025-05-30T21:34:16.028ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 87.337503ms iteration = 49 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
34472025-05-30T21:34:16.028ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34482025-05-30T21:34:16.029ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
34492025-05-30T21:34:16.029ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
34502025-05-30T21:34:16.029ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 88.173853ms iteration = 9 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
34512025-05-30T21:34:16.029ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1590, tv_nsec: 966695614 } })
34522025-05-30T21:34:16.029ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 88.555607ms iteration = 17 status = {"errors":[],"garbage_collect_requested":[]}
34532025-05-30T21:34:16.029ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
34542025-05-30T21:34:16.029ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34552025-05-30T21:34:16.032ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
34562025-05-30T21:34:16.033ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
34572025-05-30T21:34:16.033ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34582025-05-30T21:34:16.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34592025-05-30T21:34:16.033ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
34602025-05-30T21:34:16.033ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
34612025-05-30T21:34:16.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
34622025-05-30T21:34:16.051ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
34632025-05-30T21:34:16.051ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 110.661761ms iteration = 49 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
34642025-05-30T21:34:16.059ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
34652025-05-30T21:34:16.059ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
34662025-05-30T21:34:16.059ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 118.827804ms iteration = 9 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
34672025-05-30T21:34:16.069ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
34682025-05-30T21:34:16.069ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
34692025-05-30T21:34:16.069ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
34702025-05-30T21:34:16.069ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
34712025-05-30T21:34:16.069ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
34722025-05-30T21:34:16.069ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 129.254364ms iteration = 17 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
34732025-05-30T21:34:16.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
34742025-05-30T21:34:16.072ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34752025-05-30T21:34:16.073ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34762025-05-30T21:34:16.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34772025-05-30T21:34:16.074ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
34782025-05-30T21:34:16.074ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
34792025-05-30T21:34:16.074ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3b47e0e-0d5e-4c1f-b9a7-2ab2d63130d8", "content-length": "26", "date": "Fri, 30 May 2025 21:34:16 GMT"} })
34802025-05-30T21:34:16.074ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
34812025-05-30T21:34:16.076ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
34822025-05-30T21:34:16.076ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
34832025-05-30T21:34:16.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
34842025-05-30T21:34:16.076ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34852025-05-30T21:34:16.076ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
34862025-05-30T21:34:16.076ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
34872025-05-30T21:34:16.076ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
34882025-05-30T21:34:16.076ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 136.237915ms iteration = 17 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
34892025-05-30T21:34:16.077ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34902025-05-30T21:34:16.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34912025-05-30T21:34:16.079ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
34922025-05-30T21:34:16.079ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34932025-05-30T21:34:16.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34942025-05-30T21:34:16.079ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34952025-05-30T21:34:16.082ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
34962025-05-30T21:34:16.082ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34972025-05-30T21:34:16.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34982025-05-30T21:34:16.083ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
34992025-05-30T21:34:16.083ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
35002025-05-30T21:34:16.084ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc1fb5f0-10b0-4de0-90c9-e5148c89233b", "content-length": "26", "date": "Fri, 30 May 2025 21:34:16 GMT"} })
35012025-05-30T21:34:16.084ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
35022025-05-30T21:34:16.084ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
35032025-05-30T21:34:16.084ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
35042025-05-30T21:34:16.084ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
35052025-05-30T21:34:16.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
35062025-05-30T21:34:16.085ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 144.5126ms iteration = 17 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
35072025-05-30T21:34:16.085ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 144.666022ms iteration = 17 status = 0
35082025-05-30T21:34:16.085ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
35092025-05-30T21:34:16.086ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35102025-05-30T21:34:16.090ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35112025-05-30T21:34:16.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35122025-05-30T21:34:16.091ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
35132025-05-30T21:34:16.091ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
35142025-05-30T21:34:16.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
35152025-05-30T21:34:16.095ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
35162025-05-30T21:34:16.095ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
35172025-05-30T21:34:16.095ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-30 21:26:17.226115 UTC dns_group = internal generation = 1
35182025-05-30T21:34:16.095ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 155.878811ms iteration = 9 status = {"generation":1}
35192025-05-30T21:34:16.104ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35202025-05-30T21:34:16.107ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
35212025-05-30T21:34:16.107ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35222025-05-30T21:34:16.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35232025-05-30T21:34:16.134ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
35242025-05-30T21:34:16.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
35252025-05-30T21:34:16.150ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35262025-05-30T21:34:16.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35272025-05-30T21:34:16.151ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
35282025-05-30T21:34:16.151ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 0 version = 2
35292025-05-30T21:34:16.151ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
35302025-05-30T21:34:16.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
35312025-05-30T21:34:16.153ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
35322025-05-30T21:34:16.153ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 212.761064ms iteration = 17 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
35332025-05-30T21:34:16.153ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35342025-05-30T21:34:16.156ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
35352025-05-30T21:34:16.156ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
35362025-05-30T21:34:16.156ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
35372025-05-30T21:34:16.158ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35382025-05-30T21:34:16.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35392025-05-30T21:34:16.191ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-30T21:26:13.986396Z, time_modified: 2025-05-30T21:26:13.986396Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }) }
35402025-05-30T21:34:16.191ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 250.457296ms iteration = 9 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]}},"default_endpoint":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
35412025-05-30T21:34:16.209ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 268.593275ms iteration = 18 status = {"physical_disks_added":0}
35422025-05-30T21:34:16.226ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 285.938354ms iteration = 17 status = {}
35432025-05-30T21:34:16.235ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
35442025-05-30T21:34:16.235ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "fdf1059b-7cdf-442e-9273-5d80adfe5027", "date": "Fri, 30 May 2025 21:34:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
35452025-05-30T21:34:16.235ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
35462025-05-30T21:34:16.237ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "cc8af028-55a2-49dd-a218-dd6fd4e15375", "date": "Fri, 30 May 2025 21:34:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
35472025-05-30T21:34:16.237ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
35482025-05-30T21:34:16.239ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
35492025-05-30T21:34:16.239ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 299.008834ms iteration = 11 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
35502025-05-30T21:34:16.256ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
35512025-05-30T21:34:16.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
35522025-05-30T21:34:16.257ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
35532025-05-30T21:34:16.257ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
35542025-05-30T21:34:16.257ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2eff75f-8764-44da-a41e-4d3d585806ea", "content-length": "26", "date": "Fri, 30 May 2025 21:34:16 GMT"} })
35552025-05-30T21:34:16.257ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
35562025-05-30T21:34:16.259ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
35572025-05-30T21:34:16.262ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
35582025-05-30T21:34:16.262ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
35592025-05-30T21:34:16.263ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "690070df-d1dc-49dc-b59c-235101e77c07", "content-length": "372", "date": "Fri, 30 May 2025 21:34:15 GMT"} })
35602025-05-30T21:34:16.263ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35612025-05-30T21:34:16.263ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35622025-05-30T21:34:16.264ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 1 version = 2
35632025-05-30T21:34:16.264ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
35642025-05-30T21:34:16.264ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-30 21:26:18.957109 UTC dns_group = external generation = 2
35652025-05-30T21:34:16.264ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 323.91269ms iteration = 9 status = {"generation":2}
35662025-05-30T21:34:16.265ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35672025-05-30T21:34:16.266ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 325.300017ms iteration = 17 status = null
35682025-05-30T21:34:16.267ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
35692025-05-30T21:34:16.267ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35702025-05-30T21:34:16.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35712025-05-30T21:34:16.268ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35722025-05-30T21:34:16.270ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
35732025-05-30T21:34:16.270ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35742025-05-30T21:34:16.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35752025-05-30T21:34:16.277ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
35762025-05-30T21:34:16.292ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
35772025-05-30T21:34:16.292ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 351.964743ms iteration = 9 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
35782025-05-30T21:34:16.312ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
35792025-05-30T21:34:16.312ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
35802025-05-30T21:34:16.312ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "f208eb65-9db5-4fe3-9e16-458842d677fa", "date": "Fri, 30 May 2025 21:34:15 GMT"} })
35812025-05-30T21:34:16.312ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35822025-05-30T21:34:16.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35832025-05-30T21:34:16.315ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35842025-05-30T21:34:16.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35852025-05-30T21:34:16.317ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35862025-05-30T21:34:16.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35872025-05-30T21:34:16.320ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
35882025-05-30T21:34:16.320ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
35892025-05-30T21:34:16.321ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "ec85d730-133d-45c9-9b6b-b3775ac832a6", "date": "Fri, 30 May 2025 21:34:15 GMT"} })
35902025-05-30T21:34:16.322ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 381.888117ms iteration = 17 status = {}
35912025-05-30T21:34:16.470ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
35922025-05-30T21:34:16.470ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0022b116-fb7e-41a8-8a43-0a8793f451ba", "content-length": "691", "date": "Fri, 30 May 2025 21:34:15 GMT"} })
35932025-05-30T21:34:16.470ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
35942025-05-30T21:34:16.470ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
35952025-05-30T21:34:16.470ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
35962025-05-30T21:34:16.471ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 530.904768ms iteration = 17 status = {}
35972025-05-30T21:34:16.480ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
35982025-05-30T21:34:16.480ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "0443f87b-7978-4e98-9011-626ed8d939cb", "date": "Fri, 30 May 2025 21:34:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
35992025-05-30T21:34:16.480ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
36002025-05-30T21:34:16.481ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "d08b0313-86c0-4f4a-8fca-29a57ad592a9", "date": "Fri, 30 May 2025 21:34:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
36012025-05-30T21:34:16.481ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
36022025-05-30T21:34:16.483ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "d2413992-1886-4cfa-b3b0-1319668b4faa", "date": "Fri, 30 May 2025 21:34:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
36032025-05-30T21:34:16.483ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
36042025-05-30T21:34:16.485ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
36052025-05-30T21:34:16.485ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 544.687326ms iteration = 11 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
36062025-05-30T21:34:16.500ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36072025-05-30T21:34:16.500ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36082025-05-30T21:34:16.502ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
36092025-05-30T21:34:16.502ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36102025-05-30T21:34:16.503ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36112025-05-30T21:34:16.512ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
36122025-05-30T21:34:16.512ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
36132025-05-30T21:34:16.512ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
36142025-05-30T21:34:16.512ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
36152025-05-30T21:34:16.512ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36162025-05-30T21:34:16.512ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36172025-05-30T21:34:16.512ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36182025-05-30T21:34:16.512ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
36192025-05-30T21:34:16.513ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36202025-05-30T21:34:16.513ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
36212025-05-30T21:34:16.513ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
36222025-05-30T21:34:16.513ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36232025-05-30T21:34:16.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36242025-05-30T21:34:16.515ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
36252025-05-30T21:34:16.515ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
36262025-05-30T21:34:16.620ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "aa14eb4e-e08b-46f2-ab88-e2fb2696970b", "date": "Fri, 30 May 2025 21:34:16 GMT"} })
36272025-05-30T21:34:16.620ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
36282025-05-30T21:34:16.620ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
36292025-05-30T21:34:16.620ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "24cd6b8b-1320-498f-b75d-02d0c84997d2", "content-length": "682", "date": "Fri, 30 May 2025 21:34:16 GMT"} })
36302025-05-30T21:34:16.620ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36312025-05-30T21:34:16.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36322025-05-30T21:34:16.624ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36332025-05-30T21:34:16.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36342025-05-30T21:34:16.626ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36352025-05-30T21:34:16.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36362025-05-30T21:34:16.628ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36372025-05-30T21:34:16.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36382025-05-30T21:34:16.635ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 694.481647ms iteration = 17 status = {}
36392025-05-30T21:34:18.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 320 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 2e978690-13a3-475b-928d-5f853c477a32 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
36402025-05-30T21:34:25.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
36412025-05-30T21:34:25.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 50 reason = Timeout
36422025-05-30T21:34:25.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36432025-05-30T21:34:25.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36442025-05-30T21:34:25.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36452025-05-30T21:34:25.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36462025-05-30T21:34:25.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36472025-05-30T21:34:25.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36482025-05-30T21:34:25.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36492025-05-30T21:34:25.945ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.32821ms iteration = 50 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
36502025-05-30T21:34:25.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
36512025-05-30T21:34:25.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36522025-05-30T21:34:25.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36532025-05-30T21:34:25.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36542025-05-30T21:34:25.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36552025-05-30T21:34:25.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36562025-05-30T21:34:25.953ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
36572025-05-30T21:34:25.953ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36582025-05-30T21:34:25.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36592025-05-30T21:34:25.961ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
36602025-05-30T21:34:25.961ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.130441ms iteration = 50 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
36612025-05-30T21:34:28.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 348 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 611b1eb0-1247-47ee-8467-f79d858e299c response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
36622025-05-30T21:34:35.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
36632025-05-30T21:34:35.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 51 reason = Timeout
36642025-05-30T21:34:35.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36652025-05-30T21:34:35.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36662025-05-30T21:34:35.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36672025-05-30T21:34:35.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36682025-05-30T21:34:35.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36692025-05-30T21:34:35.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36702025-05-30T21:34:35.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36712025-05-30T21:34:35.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
36722025-05-30T21:34:35.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36732025-05-30T21:34:35.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36742025-05-30T21:34:35.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36752025-05-30T21:34:35.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36762025-05-30T21:34:35.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36772025-05-30T21:34:35.952ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
36782025-05-30T21:34:35.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36792025-05-30T21:34:35.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36802025-05-30T21:34:35.963ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
36812025-05-30T21:34:35.963ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.076553ms iteration = 51 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
36822025-05-30T21:34:35.977ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 36.068864ms iteration = 51 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
36832025-05-30T21:34:38.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 344 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = f66294df-ebd1-4f91-9886-56ca545e1698 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
36842025-05-30T21:34:43.766ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 3ea9c3a2-d92e-41d2-8473-b49674d04ca7 roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
36852025-05-30T21:34:43.766ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 3ea9c3a2-d92e-41d2-8473-b49674d04ca7 resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
36862025-05-30T21:34:43.768ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2670 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 3ea9c3a2-d92e-41d2-8473-b49674d04ca7 response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
36872025-05-30T21:34:43.768ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = d24dc16e-f618-4624-bb3e-7cc0447269ab roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
36882025-05-30T21:34:43.769ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = d24dc16e-f618-4624-bb3e-7cc0447269ab resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
36892025-05-30T21:34:43.770ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 1986 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = d24dc16e-f618-4624-bb3e-7cc0447269ab response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
36902025-05-30T21:34:45.940ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 52 reason = Timeout
36912025-05-30T21:34:45.940ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 18 reason = Timeout
36922025-05-30T21:34:45.940ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36932025-05-30T21:34:45.940ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
36942025-05-30T21:34:45.940ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
36952025-05-30T21:34:45.940ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
36962025-05-30T21:34:45.940ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 18 reason = Timeout
36972025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
36982025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 18 reason = Timeout
36992025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
37002025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
37012025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 19 reason = Timeout
37022025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37032025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37042025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
37052025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
37062025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 18 reason = Timeout
37072025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37082025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
37092025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 18 reason = Timeout
37102025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37112025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
37122025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 18 reason = Timeout
37132025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
37142025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 18 reason = Timeout
37152025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 18 reason = Timeout
37162025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
37172025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
37182025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
37192025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
37202025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 18 reason = Timeout
37212025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
37222025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
37232025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 18 reason = Timeout
37242025-05-30T21:34:45.941ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
37252025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37262025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
37272025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 18 reason = Timeout
37282025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
37292025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
37302025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
37312025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
37322025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
37332025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
37342025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37352025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
37362025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
37372025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
37382025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
37392025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
37402025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
37412025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37422025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
37432025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
37442025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37452025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
37462025-05-30T21:34:45.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37472025-05-30T21:34:45.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
37482025-05-30T21:34:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
37492025-05-30T21:34:45.944ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.145316ms iteration = 18 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
37502025-05-30T21:34:45.945ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1620, tv_nsec: 882592261 } })
37512025-05-30T21:34:45.945ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
37522025-05-30T21:34:45.945ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1620, tv_nsec: 882968445 } })
37532025-05-30T21:34:45.946ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37542025-05-30T21:34:45.946ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
37552025-05-30T21:34:45.946ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37562025-05-30T21:34:45.947ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37572025-05-30T21:34:45.949ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
37582025-05-30T21:34:45.949ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37592025-05-30T21:34:45.949ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
37602025-05-30T21:34:45.949ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37612025-05-30T21:34:45.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37622025-05-30T21:34:45.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
37632025-05-30T21:34:45.949ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
37642025-05-30T21:34:45.949ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
37652025-05-30T21:34:45.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
37662025-05-30T21:34:45.951ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37672025-05-30T21:34:45.952ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37682025-05-30T21:34:45.952ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
37692025-05-30T21:34:45.952ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
37702025-05-30T21:34:45.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
37712025-05-30T21:34:45.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
37722025-05-30T21:34:45.955ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37732025-05-30T21:34:45.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37742025-05-30T21:34:45.958ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37752025-05-30T21:34:45.959ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37762025-05-30T21:34:45.961ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37772025-05-30T21:34:45.961ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
37782025-05-30T21:34:45.961ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37792025-05-30T21:34:45.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37802025-05-30T21:34:45.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37812025-05-30T21:34:45.961ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37822025-05-30T21:34:45.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
37832025-05-30T21:34:45.962ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
37842025-05-30T21:34:45.962ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37852025-05-30T21:34:45.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37862025-05-30T21:34:45.964ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 23.176517ms iteration = 18 status = {"errors":[],"requests_created_ok":[]}
37872025-05-30T21:34:45.965ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37882025-05-30T21:34:45.966ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1620, tv_nsec: 903293789 } })
37892025-05-30T21:34:45.966ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37902025-05-30T21:34:45.966ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
37912025-05-30T21:34:45.967ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37922025-05-30T21:34:45.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37932025-05-30T21:34:45.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37942025-05-30T21:34:45.969ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
37952025-05-30T21:34:45.969ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37962025-05-30T21:34:45.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37972025-05-30T21:34:45.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
37982025-05-30T21:34:45.969ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37992025-05-30T21:34:45.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
38002025-05-30T21:34:45.971ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
38012025-05-30T21:34:45.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38022025-05-30T21:34:45.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38032025-05-30T21:34:45.982ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
38042025-05-30T21:34:45.982ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 41.260023ms iteration = 52 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
38052025-05-30T21:34:45.986ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 45.421162ms iteration = 18 status = {"errors":[],"finish_invoked_ok":[]}
38062025-05-30T21:34:45.997ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
38072025-05-30T21:34:45.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
38082025-05-30T21:34:45.998ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
38092025-05-30T21:34:45.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
38102025-05-30T21:34:45.999ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 58.710514ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
38112025-05-30T21:34:46.000ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
38122025-05-30T21:34:46.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
38132025-05-30T21:34:46.001ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
38142025-05-30T21:34:46.001ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
38152025-05-30T21:34:46.001ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
38162025-05-30T21:34:46.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
38172025-05-30T21:34:46.002ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 61.012211ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
38182025-05-30T21:34:46.002ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 61.390815ms iteration = 18 status = {"errors":[],"garbage_collect_requested":[]}
38192025-05-30T21:34:46.003ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.209004ms iteration = 52 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
38202025-05-30T21:34:46.003ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
38212025-05-30T21:34:46.003ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 62.143314ms iteration = 18 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
38222025-05-30T21:34:46.003ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38232025-05-30T21:34:46.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38242025-05-30T21:34:46.006ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38252025-05-30T21:34:46.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38262025-05-30T21:34:46.007ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38272025-05-30T21:34:46.010ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38282025-05-30T21:34:46.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38292025-05-30T21:34:46.010ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
38302025-05-30T21:34:46.010ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38312025-05-30T21:34:46.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38322025-05-30T21:34:46.012ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38332025-05-30T21:34:46.013ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 72.25574ms iteration = 18 status = 0
38342025-05-30T21:34:46.014ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
38352025-05-30T21:34:46.014ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38362025-05-30T21:34:46.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38372025-05-30T21:34:46.014ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 73.877119ms iteration = 19 status = {"physical_disks_added":0}
38382025-05-30T21:34:46.022ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
38392025-05-30T21:34:46.022ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
38402025-05-30T21:34:46.022ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "aef13036-6ef9-452e-a0b4-03c76ca26f79", "content-length": "26", "date": "Fri, 30 May 2025 21:34:45 GMT"} })
38412025-05-30T21:34:46.022ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
38422025-05-30T21:34:46.025ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
38432025-05-30T21:34:46.027ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
38442025-05-30T21:34:46.027ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
38452025-05-30T21:34:46.027ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f598410e-99fe-4c50-9ae7-a26111a0ede1", "content-length": "26", "date": "Fri, 30 May 2025 21:34:45 GMT"} })
38462025-05-30T21:34:46.027ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
38472025-05-30T21:34:46.029ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
38482025-05-30T21:34:46.033ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
38492025-05-30T21:34:46.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
38502025-05-30T21:34:46.044ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
38512025-05-30T21:34:46.044ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
38522025-05-30T21:34:46.044ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
38532025-05-30T21:34:46.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
38542025-05-30T21:34:46.045ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
38552025-05-30T21:34:46.045ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
38562025-05-30T21:34:46.045ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef87b572-2166-4c2a-8022-421a2bec2d50", "content-length": "26", "date": "Fri, 30 May 2025 21:34:45 GMT"} })
38572025-05-30T21:34:46.045ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
38582025-05-30T21:34:46.047ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
38592025-05-30T21:34:46.048ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
38602025-05-30T21:34:46.048ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 107.861649ms iteration = 18 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
38612025-05-30T21:34:46.084ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 143.308005ms iteration = 18 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
38622025-05-30T21:34:46.095ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
38632025-05-30T21:34:46.095ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
38642025-05-30T21:34:46.095ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 154.655525ms iteration = 18 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
38652025-05-30T21:34:46.104ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38662025-05-30T21:34:46.104ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38672025-05-30T21:34:46.120ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 179.652783ms iteration = 18 status = {}
38682025-05-30T21:34:46.135ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
38692025-05-30T21:34:46.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
38702025-05-30T21:34:46.137ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 196.86129ms iteration = 18 status = null
38712025-05-30T21:34:46.146ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38722025-05-30T21:34:46.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38732025-05-30T21:34:46.148ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38742025-05-30T21:34:46.150ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
38752025-05-30T21:34:46.150ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38762025-05-30T21:34:46.150ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38772025-05-30T21:34:46.151ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38782025-05-30T21:34:46.151ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
38792025-05-30T21:34:46.151ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
38802025-05-30T21:34:46.152ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b9ebaa5-6dea-44c0-bc32-6df7afb86438", "content-length": "372", "date": "Fri, 30 May 2025 21:34:45 GMT"} })
38812025-05-30T21:34:46.152ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38822025-05-30T21:34:46.152ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38832025-05-30T21:34:46.153ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
38842025-05-30T21:34:46.153ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38852025-05-30T21:34:46.153ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38862025-05-30T21:34:46.169ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
38872025-05-30T21:34:46.169ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
38882025-05-30T21:34:46.170ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "62d07bad-1fbf-4b41-84e6-8be493f0ce96", "date": "Fri, 30 May 2025 21:34:45 GMT"} })
38892025-05-30T21:34:46.170ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38902025-05-30T21:34:46.171ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38912025-05-30T21:34:46.173ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38922025-05-30T21:34:46.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38932025-05-30T21:34:46.177ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38942025-05-30T21:34:46.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38952025-05-30T21:34:46.181ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
38962025-05-30T21:34:46.181ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
38972025-05-30T21:34:46.181ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "dc276777-09ff-4908-99b1-d4f8c8fce7f3", "date": "Fri, 30 May 2025 21:34:45 GMT"} })
38982025-05-30T21:34:46.183ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 242.176881ms iteration = 18 status = {}
38992025-05-30T21:34:46.207ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39002025-05-30T21:34:46.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39012025-05-30T21:34:46.209ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
39022025-05-30T21:34:46.209ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39032025-05-30T21:34:46.209ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39042025-05-30T21:34:46.222ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
39052025-05-30T21:34:46.222ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
39062025-05-30T21:34:46.222ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
39072025-05-30T21:34:46.222ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
39082025-05-30T21:34:46.222ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39092025-05-30T21:34:46.222ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39102025-05-30T21:34:46.222ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39112025-05-30T21:34:46.222ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
39122025-05-30T21:34:46.223ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39132025-05-30T21:34:46.223ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
39142025-05-30T21:34:46.223ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
39152025-05-30T21:34:46.223ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39162025-05-30T21:34:46.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39172025-05-30T21:34:46.225ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
39182025-05-30T21:34:46.225ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
39192025-05-30T21:34:46.247ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
39202025-05-30T21:34:46.336ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "b2c69e46-c93d-4447-8d35-7655afb8c1df", "date": "Fri, 30 May 2025 21:34:45 GMT"} })
39212025-05-30T21:34:46.336ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
39222025-05-30T21:34:46.336ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
39232025-05-30T21:34:46.336ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0d68f3b-86f0-4aa1-9925-0ffcdf45751a", "content-length": "691", "date": "Fri, 30 May 2025 21:34:45 GMT"} })
39242025-05-30T21:34:46.336ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
39252025-05-30T21:34:46.336ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
39262025-05-30T21:34:46.337ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
39272025-05-30T21:34:46.337ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cd5b82e-d73f-4718-bae9-ab7471c20648", "content-length": "682", "date": "Fri, 30 May 2025 21:34:45 GMT"} })
39282025-05-30T21:34:46.337ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39292025-05-30T21:34:46.337ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39302025-05-30T21:34:46.338ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 397.375823ms iteration = 18 status = {}
39312025-05-30T21:34:46.341ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39322025-05-30T21:34:46.341ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39332025-05-30T21:34:46.345ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39342025-05-30T21:34:46.345ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39352025-05-30T21:34:46.347ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39362025-05-30T21:34:46.348ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39372025-05-30T21:34:46.355ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 414.170886ms iteration = 18 status = {}
39382025-05-30T21:34:46.591ZDEBGnexus (producer-registration-task): registering as oximeter producer with Nexus registration_count = 3
39392025-05-30T21:34:46.591ZDEBGnexus (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221
39402025-05-30T21:34:46.620ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
39412025-05-30T21:34:46.620ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:62074
39422025-05-30T21:34:46.621ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:62074 req_id = 8eef360a-69b7-4adf-ae9d-20288cf9ded9 roles = RoleSet { roles: {} } uri = /metrics/producers
39432025-05-30T21:34:46.621ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:62074 req_id = 8eef360a-69b7-4adf-ae9d-20288cf9ded9 resource = Database result = Ok(()) uri = /metrics/producers
39442025-05-30T21:34:46.661ZINFOnexus: assigned collector to new producer collector_id = bc707af4-d9a3-402f-b2b6-642b29342fa8 file = nexus/src/app/oximeter.rs:76 producer_id = bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
39452025-05-30T21:34:46.661ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 40873 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:62074 req_id = 8eef360a-69b7-4adf-ae9d-20288cf9ded9 response_code = 201 uri = /metrics/producers
39462025-05-30T21:34:46.661ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "8eef360a-69b7-4adf-ae9d-20288cf9ded9", "content-length": "41", "date": "Fri, 30 May 2025 21:34:46 GMT"} })
39472025-05-30T21:34:46.663ZDEBGnexus (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
39482025-05-30T21:34:48.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 345 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 75692b94-f083-4e12-95cf-d47f911b3da9 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
39492025-05-30T21:34:55.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 53 reason = Timeout
39502025-05-30T21:34:55.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
39512025-05-30T21:34:55.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
39522025-05-30T21:34:55.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39532025-05-30T21:34:55.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
39542025-05-30T21:34:55.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39552025-05-30T21:34:55.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39562025-05-30T21:34:55.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
39572025-05-30T21:34:55.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39582025-05-30T21:34:55.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39592025-05-30T21:34:55.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39602025-05-30T21:34:55.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39612025-05-30T21:34:55.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39622025-05-30T21:34:55.953ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
39632025-05-30T21:34:55.953ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39642025-05-30T21:34:55.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39652025-05-30T21:34:55.965ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
39662025-05-30T21:34:55.965ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.450739ms iteration = 53 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
39672025-05-30T21:34:55.975ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
39682025-05-30T21:34:55.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
39692025-05-30T21:34:56.006ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.510812ms iteration = 53 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
39702025-05-30T21:34:58.777ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 328 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 11151b3b-5408-41ee-a461-b6a4e3d88c9d response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
39712025-05-30T21:35:05.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 54 reason = Timeout
39722025-05-30T21:35:05.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
39732025-05-30T21:35:05.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 54 reason = Timeout
39742025-05-30T21:35:05.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39752025-05-30T21:35:05.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
39762025-05-30T21:35:05.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39772025-05-30T21:35:05.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
39782025-05-30T21:35:05.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
39792025-05-30T21:35:05.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39802025-05-30T21:35:05.944ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.660162ms iteration = 54 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
39812025-05-30T21:35:05.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
39822025-05-30T21:35:05.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39832025-05-30T21:35:05.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39842025-05-30T21:35:05.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39852025-05-30T21:35:05.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39862025-05-30T21:35:05.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39872025-05-30T21:35:05.952ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
39882025-05-30T21:35:05.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39892025-05-30T21:35:05.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39902025-05-30T21:35:05.963ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
39912025-05-30T21:35:05.963ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.119654ms iteration = 54 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
39922025-05-30T21:35:08.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 355 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 26ac851c-1bd9-4fca-9db3-9f5f6cac42f0 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
39932025-05-30T21:35:13.766ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = eabec717-da08-49a8-bf7c-dc02e642c855 roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
39942025-05-30T21:35:13.767ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = eabec717-da08-49a8-bf7c-dc02e642c855 resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
39952025-05-30T21:35:13.769ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3117 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = eabec717-da08-49a8-bf7c-dc02e642c855 response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
39962025-05-30T21:35:13.770ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 5a9edd48-c1c2-4196-bdcf-495dac682916 roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
39972025-05-30T21:35:13.770ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 5a9edd48-c1c2-4196-bdcf-495dac682916 resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
39982025-05-30T21:35:13.772ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2189 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 5a9edd48-c1c2-4196-bdcf-495dac682916 response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
39992025-05-30T21:35:15.940ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
40002025-05-30T21:35:15.940ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
40012025-05-30T21:35:15.940ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
40022025-05-30T21:35:15.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
40032025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 55 reason = Timeout
40042025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40052025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
40062025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 19 reason = Timeout
40072025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 10 reason = Timeout
40082025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
40092025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
40102025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
40112025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 10 reason = Timeout
40122025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
40132025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
40142025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 10 reason = Timeout
40152025-05-30T21:35:15.941ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
40162025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 19 reason = Timeout
40172025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
40182025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 19 reason = Timeout
40192025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
40202025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
40212025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
40222025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
40232025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
40242025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 19 reason = Timeout
40252025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
40262025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
40272025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 19 reason = Timeout
40282025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 19 reason = Timeout
40292025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
40302025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 11 reason = Timeout
40312025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40322025-05-30T21:35:15.941ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e
40332025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
40342025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.88\u{b5}s iteration = 11 status = {"enabled":false,"target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e"}
40352025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 19 reason = Timeout
40362025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 5 reason = Timeout
40372025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
40382025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
40392025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
40402025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
40412025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
40422025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 55 reason = Timeout
40432025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40442025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 10 reason = Timeout
40452025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40462025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 10 reason = Timeout
40472025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
40482025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
40492025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 19 reason = Timeout
40502025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
40512025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 19 reason = Timeout
40522025-05-30T21:35:15.941ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
40532025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 10 reason = Timeout
40542025-05-30T21:35:15.941ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-30 21:25:15.941545532 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
40552025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
40562025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40572025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 10 reason = Timeout
40582025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 19 reason = Timeout
40592025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
40602025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 19 reason = Timeout
40612025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
40622025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 19 reason = Timeout
40632025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
40642025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 19 reason = Timeout
40652025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 19 reason = Timeout
40662025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
40672025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
40682025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 19 reason = Timeout
40692025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
40702025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 19 reason = Timeout
40712025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
40722025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 20 reason = Timeout
40732025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
40742025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
40752025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
40762025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
40772025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
40782025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
40792025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
40802025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
40812025-05-30T21:35:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
40822025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
40832025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
40842025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
40852025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
40862025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
40872025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40882025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
40892025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40902025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
40912025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
40922025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
40932025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
40942025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
40952025-05-30T21:35:15.942ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0413f6b4-86a1-42a1-919a-e8f42ff9f638.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("acb1452d-0a02-4b36-a462-c8ce2f96a488.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:2::1))) }], valid_until: Instant { tv_sec: 1650, tv_nsec: 879097860 } })
40962025-05-30T21:35:15.942ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2a161087-f464-41f8-b020-0f9e83fd5351.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1650, tv_nsec: 879100509 } })
40972025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
40982025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
40992025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
41002025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41012025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
41022025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
41032025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
41042025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41052025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
41062025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
41072025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
41082025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
41092025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
41102025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.242944ms iteration = 10 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
41112025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } dns_group = internal
41122025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.424837ms iteration = 10 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353"]}
41132025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
41142025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
41152025-05-30T21:35:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
41162025-05-30T21:35:15.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41172025-05-30T21:35:15.945ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41182025-05-30T21:35:15.945ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.998016ms iteration = 19 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
41192025-05-30T21:35:15.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
41202025-05-30T21:35:15.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41212025-05-30T21:35:15.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41222025-05-30T21:35:15.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
41232025-05-30T21:35:15.949ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1650, tv_nsec: 886254612 } })
41242025-05-30T21:35:15.949ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
41252025-05-30T21:35:15.950ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41262025-05-30T21:35:15.950ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41272025-05-30T21:35:15.950ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
41282025-05-30T21:35:15.950ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41292025-05-30T21:35:15.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
41302025-05-30T21:35:15.953ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
41312025-05-30T21:35:15.953ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
41322025-05-30T21:35:15.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
41332025-05-30T21:35:15.953ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
41342025-05-30T21:35:15.953ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
41352025-05-30T21:35:15.953ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
41362025-05-30T21:35:15.953ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
41372025-05-30T21:35:15.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
41382025-05-30T21:35:15.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
41392025-05-30T21:35:15.955ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1650, tv_nsec: 892867148 } })
41402025-05-30T21:35:15.956ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
41412025-05-30T21:35:15.956ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-30 21:25:15.941545532 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
41422025-05-30T21:35:15.956ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 15.104984ms iteration = 10 status = {"expiration":"2025-05-30T21:25:15.941545532Z","pruned":[]}
41432025-05-30T21:35:15.956ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41442025-05-30T21:35:15.957ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41452025-05-30T21:35:15.959ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
41462025-05-30T21:35:15.959ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41472025-05-30T21:35:15.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41482025-05-30T21:35:15.959ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
41492025-05-30T21:35:15.959ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41502025-05-30T21:35:15.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41512025-05-30T21:35:15.960ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 18.484842ms iteration = 10 status = {"errors":[],"found_port_ok":[]}
41522025-05-30T21:35:15.960ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
41532025-05-30T21:35:15.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
41542025-05-30T21:35:15.961ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41552025-05-30T21:35:15.962ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:202 num_receivers = 0
41562025-05-30T21:35:15.962ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 20.792669ms iteration = 10 status = {"by_rx":{},"error":null}
41572025-05-30T21:35:15.963ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41582025-05-30T21:35:15.964ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41592025-05-30T21:35:15.965ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
41602025-05-30T21:35:15.965ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
41612025-05-30T21:35:15.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
41622025-05-30T21:35:15.965ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
41632025-05-30T21:35:15.965ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
41642025-05-30T21:35:15.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
41652025-05-30T21:35:15.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
41662025-05-30T21:35:15.968ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41672025-05-30T21:35:15.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41682025-05-30T21:35:15.970ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41692025-05-30T21:35:15.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
41702025-05-30T21:35:15.973ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
41712025-05-30T21:35:15.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
41722025-05-30T21:35:15.974ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41732025-05-30T21:35:15.976ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
41742025-05-30T21:35:15.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
41752025-05-30T21:35:15.978ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
41762025-05-30T21:35:15.978ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
41772025-05-30T21:35:15.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
41782025-05-30T21:35:15.978ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41792025-05-30T21:35:15.980ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
41802025-05-30T21:35:15.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
41812025-05-30T21:35:15.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41822025-05-30T21:35:15.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41832025-05-30T21:35:15.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
41842025-05-30T21:35:15.981ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
41852025-05-30T21:35:15.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
41862025-05-30T21:35:15.981ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41872025-05-30T21:35:15.985ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
41882025-05-30T21:35:15.985ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
41892025-05-30T21:35:15.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
41902025-05-30T21:35:15.986ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41912025-05-30T21:35:15.988ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
41922025-05-30T21:35:15.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
41932025-05-30T21:35:15.990ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 49.076354ms iteration = 19 status = {"errors":[],"requests_created_ok":[]}
41942025-05-30T21:35:15.990ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
41952025-05-30T21:35:15.992ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
41962025-05-30T21:35:15.992ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41972025-05-30T21:35:15.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
41982025-05-30T21:35:15.994ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 52.513733ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
41992025-05-30T21:35:15.994ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1650, tv_nsec: 931236899 } })
42002025-05-30T21:35:15.994ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
42012025-05-30T21:35:15.994ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
42022025-05-30T21:35:15.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
42032025-05-30T21:35:15.995ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42042025-05-30T21:35:15.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42052025-05-30T21:35:15.996ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
42062025-05-30T21:35:15.996ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42072025-05-30T21:35:15.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42082025-05-30T21:35:15.997ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
42092025-05-30T21:35:15.997ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
42102025-05-30T21:35:15.997ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
42112025-05-30T21:35:15.997ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
42122025-05-30T21:35:15.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
42132025-05-30T21:35:15.999ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42142025-05-30T21:35:15.999ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
42152025-05-30T21:35:15.999ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
42162025-05-30T21:35:15.999ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
42172025-05-30T21:35:15.999ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 58.440541ms iteration = 10 status = {"dispatched":[],"errors":[],"glob_version":"144.0.0","globs_reprocessed":{},"no_receivers":[]}
42182025-05-30T21:35:15.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
42192025-05-30T21:35:16.000ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42202025-05-30T21:35:16.002ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
42212025-05-30T21:35:16.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
42222025-05-30T21:35:16.002ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42232025-05-30T21:35:16.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
42242025-05-30T21:35:16.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42252025-05-30T21:35:16.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
42262025-05-30T21:35:16.003ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
42272025-05-30T21:35:16.003ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42282025-05-30T21:35:16.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
42292025-05-30T21:35:16.004ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42302025-05-30T21:35:16.004ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42312025-05-30T21:35:16.005ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42322025-05-30T21:35:16.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42332025-05-30T21:35:16.006ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
42342025-05-30T21:35:16.006ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
42352025-05-30T21:35:16.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
42362025-05-30T21:35:16.007ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
42372025-05-30T21:35:16.007ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
42382025-05-30T21:35:16.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
42392025-05-30T21:35:16.008ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 66.808368ms iteration = 19 status = {"errors":[],"garbage_collect_requested":[]}
42402025-05-30T21:35:16.008ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
42412025-05-30T21:35:16.008ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
42422025-05-30T21:35:16.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
42432025-05-30T21:35:16.010ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42442025-05-30T21:35:16.011ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
42452025-05-30T21:35:16.011ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
42462025-05-30T21:35:16.011ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 69.61635ms iteration = 10 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
42472025-05-30T21:35:16.011ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
42482025-05-30T21:35:16.011ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
42492025-05-30T21:35:16.011ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
42502025-05-30T21:35:16.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
42512025-05-30T21:35:16.011ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 70.319777ms iteration = 55 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
42522025-05-30T21:35:16.012ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
42532025-05-30T21:35:16.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
42542025-05-30T21:35:16.013ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
42552025-05-30T21:35:16.014ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
42562025-05-30T21:35:16.014ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
42572025-05-30T21:35:16.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
42582025-05-30T21:35:16.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
42592025-05-30T21:35:16.018ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 76.884773ms iteration = 19 status = {"errors":[],"finish_invoked_ok":[]}
42602025-05-30T21:35:16.022ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
42612025-05-30T21:35:16.022ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
42622025-05-30T21:35:16.022ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 81.207742ms iteration = 10 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
42632025-05-30T21:35:16.023ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
42642025-05-30T21:35:16.023ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 81.84261ms iteration = 55 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
42652025-05-30T21:35:16.028ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
42662025-05-30T21:35:16.028ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 86.591555ms iteration = 19 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
42672025-05-30T21:35:16.028ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
42682025-05-30T21:35:16.028ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
42692025-05-30T21:35:16.028ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
42702025-05-30T21:35:16.028ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
42712025-05-30T21:35:16.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
42722025-05-30T21:35:16.039ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42732025-05-30T21:35:16.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42742025-05-30T21:35:16.061ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
42752025-05-30T21:35:16.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
42762025-05-30T21:35:16.073ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
42772025-05-30T21:35:16.073ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
42782025-05-30T21:35:16.074ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7c53625-1ee5-4c28-9a04-cb87aad8f863", "content-length": "26", "date": "Fri, 30 May 2025 21:35:16 GMT"} })
42792025-05-30T21:35:16.074ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
42802025-05-30T21:35:16.077ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
42812025-05-30T21:35:16.080ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
42822025-05-30T21:35:16.089ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
42832025-05-30T21:35:16.108ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
42842025-05-30T21:35:16.108ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
42852025-05-30T21:35:16.109ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6b4c1a4-c45e-470e-bb80-2652435d5d9e", "content-length": "26", "date": "Fri, 30 May 2025 21:35:16 GMT"} })
42862025-05-30T21:35:16.109ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
42872025-05-30T21:35:16.112ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
42882025-05-30T21:35:16.170ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
42892025-05-30T21:35:16.170ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
42902025-05-30T21:35:16.171ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a362ac3-28be-451c-8fb5-9c95ec495d87", "content-length": "26", "date": "Fri, 30 May 2025 21:35:16 GMT"} })
42912025-05-30T21:35:16.171ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
42922025-05-30T21:35:16.173ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
42932025-05-30T21:35:16.199ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
42942025-05-30T21:35:16.199ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
42952025-05-30T21:35:16.199ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e978aa7d-5da4-437f-80be-636873a17d9e", "content-length": "372", "date": "Fri, 30 May 2025 21:35:15 GMT"} })
42962025-05-30T21:35:16.199ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42972025-05-30T21:35:16.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42982025-05-30T21:35:16.201ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42992025-05-30T21:35:16.202ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 260.619913ms iteration = 5 status = {"nsuccess":5}
43002025-05-30T21:35:16.204ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
43012025-05-30T21:35:16.204ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
43022025-05-30T21:35:16.204ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
43032025-05-30T21:35:16.204ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 262.963979ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
43042025-05-30T21:35:16.207ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43052025-05-30T21:35:16.207ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
43062025-05-30T21:35:16.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
43072025-05-30T21:35:16.210ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
43082025-05-30T21:35:16.210ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
43092025-05-30T21:35:16.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
43102025-05-30T21:35:16.246ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 304.921732ms iteration = 19 status = {}
43112025-05-30T21:35:16.256ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43122025-05-30T21:35:16.256ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
43132025-05-30T21:35:16.256ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
43142025-05-30T21:35:16.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
43152025-05-30T21:35:16.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
43162025-05-30T21:35:16.257ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
43172025-05-30T21:35:16.257ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "23d8d3d8-8cf4-4403-a504-d6849b378caa", "date": "Fri, 30 May 2025 21:35:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
43182025-05-30T21:35:16.257ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
43192025-05-30T21:35:16.258ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
43202025-05-30T21:35:16.259ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43212025-05-30T21:35:16.259ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
43222025-05-30T21:35:16.259ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
43232025-05-30T21:35:16.259ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
43242025-05-30T21:35:16.259ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "c8c2d0d7-91f0-49ed-8fea-cea48add007a", "date": "Fri, 30 May 2025 21:35:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
43252025-05-30T21:35:16.259ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
43262025-05-30T21:35:16.259ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
43272025-05-30T21:35:16.260ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
43282025-05-30T21:35:16.260ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
43292025-05-30T21:35:16.260ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
43302025-05-30T21:35:16.262ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
43312025-05-30T21:35:16.262ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
43322025-05-30T21:35:16.262ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43332025-05-30T21:35:16.262ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
43342025-05-30T21:35:16.262ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
43352025-05-30T21:35:16.262ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 321.39794ms iteration = 12 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
43362025-05-30T21:35:16.262ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
43372025-05-30T21:35:16.263ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
43382025-05-30T21:35:16.263ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 321.902006ms iteration = 19 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
43392025-05-30T21:35:16.263ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
43402025-05-30T21:35:16.265ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 323.793998ms iteration = 19 status = null
43412025-05-30T21:35:16.265ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
43422025-05-30T21:35:16.265ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
43432025-05-30T21:35:16.265ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 324.110492ms iteration = 19 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
43442025-05-30T21:35:16.265ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
43452025-05-30T21:35:16.265ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
43462025-05-30T21:35:16.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
43472025-05-30T21:35:16.268ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
43482025-05-30T21:35:16.268ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
43492025-05-30T21:35:16.268ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-30 21:26:17.226115 UTC dns_group = internal generation = 1
43502025-05-30T21:35:16.268ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 328.268749ms iteration = 10 status = {"generation":1}
43512025-05-30T21:35:16.272ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43522025-05-30T21:35:16.275ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
43532025-05-30T21:35:16.275ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
43542025-05-30T21:35:16.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
43552025-05-30T21:35:16.277ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 335.838016ms iteration = 19 status = 0
43562025-05-30T21:35:16.280ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
43572025-05-30T21:35:16.280ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
43582025-05-30T21:35:16.280ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "fbabe6cd-caf3-4823-85c2-adf0500f9ae6", "date": "Fri, 30 May 2025 21:35:15 GMT"} })
43592025-05-30T21:35:16.280ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43602025-05-30T21:35:16.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43612025-05-30T21:35:16.283ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43622025-05-30T21:35:16.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43632025-05-30T21:35:16.286ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43642025-05-30T21:35:16.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43652025-05-30T21:35:16.289ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
43662025-05-30T21:35:16.289ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
43672025-05-30T21:35:16.290ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "2050a2bd-43e1-46c0-b96d-d22466b738fe", "date": "Fri, 30 May 2025 21:35:15 GMT"} })
43682025-05-30T21:35:16.292ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 350.443384ms iteration = 19 status = {}
43692025-05-30T21:35:16.293ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 351.590658ms iteration = 19 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
43702025-05-30T21:35:16.298ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-30T21:26:13.986396Z, time_modified: 2025-05-30T21:26:13.986396Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }) }
43712025-05-30T21:35:16.298ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 357.136061ms iteration = 10 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]}},"default_endpoint":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
43722025-05-30T21:35:16.300ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
43732025-05-30T21:35:16.300ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 0 version = 2
43742025-05-30T21:35:16.300ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
43752025-05-30T21:35:16.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
43762025-05-30T21:35:16.301ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43772025-05-30T21:35:16.305ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
43782025-05-30T21:35:16.306ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "f000b750-988a-44fd-ad66-6640c73a2aa5", "date": "Fri, 30 May 2025 21:35:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
43792025-05-30T21:35:16.306ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
43802025-05-30T21:35:16.306ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
43812025-05-30T21:35:16.306ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
43822025-05-30T21:35:16.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
43832025-05-30T21:35:16.308ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "2425a0d8-c6dd-45cf-b9ba-f1be4765f860", "date": "Fri, 30 May 2025 21:35:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
43842025-05-30T21:35:16.308ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
43852025-05-30T21:35:16.309ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 1 version = 2
43862025-05-30T21:35:16.309ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
43872025-05-30T21:35:16.309ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-30 21:26:18.957109 UTC dns_group = external generation = 2
43882025-05-30T21:35:16.309ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 367.594202ms iteration = 10 status = {"generation":2}
43892025-05-30T21:35:16.310ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "d803c1f4-a9e9-4635-9245-346912ab046e", "date": "Fri, 30 May 2025 21:35:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
43902025-05-30T21:35:16.310ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
43912025-05-30T21:35:16.312ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
43922025-05-30T21:35:16.312ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 371.533587ms iteration = 12 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
43932025-05-30T21:35:16.333ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 391.933771ms iteration = 20 status = {"physical_disks_added":0}
43942025-05-30T21:35:16.365ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
43952025-05-30T21:35:16.365ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 424.09123ms iteration = 10 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
43962025-05-30T21:35:16.378ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43972025-05-30T21:35:16.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43982025-05-30T21:35:16.378ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
43992025-05-30T21:35:16.379ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "9eab4d91-5a7f-444c-bb22-64d3c259a2bf", "content-length": "691", "date": "Fri, 30 May 2025 21:35:15 GMT"} })
44002025-05-30T21:35:16.379ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
44012025-05-30T21:35:16.379ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
44022025-05-30T21:35:16.379ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
44032025-05-30T21:35:16.380ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
44042025-05-30T21:35:16.380ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44052025-05-30T21:35:16.380ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 439.438146ms iteration = 19 status = {}
44062025-05-30T21:35:16.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44072025-05-30T21:35:16.391ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
44082025-05-30T21:35:16.391ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
44092025-05-30T21:35:16.391ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
44102025-05-30T21:35:16.391ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
44112025-05-30T21:35:16.391ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
44122025-05-30T21:35:16.391ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
44132025-05-30T21:35:16.391ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
44142025-05-30T21:35:16.391ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
44152025-05-30T21:35:16.392ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
44162025-05-30T21:35:16.392ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
44172025-05-30T21:35:16.392ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
44182025-05-30T21:35:16.392ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44192025-05-30T21:35:16.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44202025-05-30T21:35:16.394ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
44212025-05-30T21:35:16.394ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
44222025-05-30T21:35:16.500ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "e5fa86e3-7bbb-47ea-9d86-c800b4883a84", "date": "Fri, 30 May 2025 21:35:15 GMT"} })
44232025-05-30T21:35:16.500ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
44242025-05-30T21:35:16.500ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
44252025-05-30T21:35:16.500ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ad4ff01-dbb9-4373-9332-e8272f6ecdcb", "content-length": "682", "date": "Fri, 30 May 2025 21:35:16 GMT"} })
44262025-05-30T21:35:16.500ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44272025-05-30T21:35:16.501ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44282025-05-30T21:35:16.504ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44292025-05-30T21:35:16.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44302025-05-30T21:35:16.508ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44312025-05-30T21:35:16.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44322025-05-30T21:35:16.510ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44332025-05-30T21:35:16.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44342025-05-30T21:35:16.517ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 576.539001ms iteration = 19 status = {}
44352025-05-30T21:35:18.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 318 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 2fa56e92-7043-4b6e-9804-363d89659248 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
44362025-05-30T21:35:25.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
44372025-05-30T21:35:25.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 56 reason = Timeout
44382025-05-30T21:35:25.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44392025-05-30T21:35:25.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44402025-05-30T21:35:25.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44412025-05-30T21:35:25.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44422025-05-30T21:35:25.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44432025-05-30T21:35:25.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
44442025-05-30T21:35:25.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44452025-05-30T21:35:25.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44462025-05-30T21:35:25.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44472025-05-30T21:35:25.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44482025-05-30T21:35:25.976ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44492025-05-30T21:35:25.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44502025-05-30T21:35:25.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44512025-05-30T21:35:25.984ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
44522025-05-30T21:35:25.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44532025-05-30T21:35:25.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44542025-05-30T21:35:25.995ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
44552025-05-30T21:35:25.995ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 53.134691ms iteration = 56 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
44562025-05-30T21:35:26.007ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 65.32169ms iteration = 56 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
44572025-05-30T21:35:28.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 314 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 336d3018-2092-4a11-b785-d1477b28f9b9 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
44582025-05-30T21:35:35.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 57 reason = Timeout
44592025-05-30T21:35:35.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44602025-05-30T21:35:35.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 57 reason = Timeout
44612025-05-30T21:35:35.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44622025-05-30T21:35:35.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44632025-05-30T21:35:35.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44642025-05-30T21:35:35.944ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44652025-05-30T21:35:35.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44662025-05-30T21:35:35.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44672025-05-30T21:35:35.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
44682025-05-30T21:35:35.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44692025-05-30T21:35:35.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44702025-05-30T21:35:35.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44712025-05-30T21:35:35.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44722025-05-30T21:35:35.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44732025-05-30T21:35:35.954ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
44742025-05-30T21:35:35.954ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44752025-05-30T21:35:35.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44762025-05-30T21:35:35.965ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
44772025-05-30T21:35:35.965ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.250867ms iteration = 57 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
44782025-05-30T21:35:35.977ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 35.598579ms iteration = 57 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
44792025-05-30T21:35:38.777ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 361 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 677137c0-caff-4b3c-8cbf-716ed3323e7e response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
44802025-05-30T21:35:43.767ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 2f3d0ff0-b5be-4e08-9d7c-0aaab8c7830a roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
44812025-05-30T21:35:43.767ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 2f3d0ff0-b5be-4e08-9d7c-0aaab8c7830a resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
44822025-05-30T21:35:43.770ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3130 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 2f3d0ff0-b5be-4e08-9d7c-0aaab8c7830a response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
44832025-05-30T21:35:43.770ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 0d575a09-cb85-4805-a728-f2ab4624a037 roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
44842025-05-30T21:35:43.770ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 0d575a09-cb85-4805-a728-f2ab4624a037 resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
44852025-05-30T21:35:43.772ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2179 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64220 req_id = 0d575a09-cb85-4805-a728-f2ab4624a037 response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
44862025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 58 reason = Timeout
44872025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44882025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 20 reason = Timeout
44892025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
44902025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 20 reason = Timeout
44912025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
44922025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 20 reason = Timeout
44932025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 20 reason = Timeout
44942025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
44952025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
44962025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
44972025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
44982025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44992025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 20 reason = Timeout
45002025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
45012025-05-30T21:35:45.941ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
45022025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
45032025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45042025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 20 reason = Timeout
45052025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 20 reason = Timeout
45062025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
45072025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
45082025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 21 reason = Timeout
45092025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
45102025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 20 reason = Timeout
45112025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
45122025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 20 reason = Timeout
45132025-05-30T21:35:45.941ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
45142025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 58 reason = Timeout
45152025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45162025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
45172025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
45182025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 20 reason = Timeout
45192025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
45202025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
45212025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
45222025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
45232025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 20 reason = Timeout
45242025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
45252025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
45262025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
45272025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
45282025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
45292025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
45302025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
45312025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45322025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
45332025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45342025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
45352025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
45362025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
45372025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45382025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
45392025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
45402025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
45412025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
45422025-05-30T21:35:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
45432025-05-30T21:35:45.945ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45442025-05-30T21:35:45.945ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
45452025-05-30T21:35:45.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
45462025-05-30T21:35:45.946ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.549532ms iteration = 20 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
45472025-05-30T21:35:45.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
45482025-05-30T21:35:45.947ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
45492025-05-30T21:35:45.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
45502025-05-30T21:35:45.947ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
45512025-05-30T21:35:45.947ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
45522025-05-30T21:35:45.947ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
45532025-05-30T21:35:45.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
45542025-05-30T21:35:45.950ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1680, tv_nsec: 886924112 } })
45552025-05-30T21:35:45.950ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
45562025-05-30T21:35:45.951ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45572025-05-30T21:35:45.953ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
45582025-05-30T21:35:45.953ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
45592025-05-30T21:35:45.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
45602025-05-30T21:35:45.953ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1680, tv_nsec: 890392382 } })
45612025-05-30T21:35:45.954ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
45622025-05-30T21:35:46.000ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
45632025-05-30T21:35:46.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
45642025-05-30T21:35:46.001ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 60.316593ms iteration = 20 status = {"errors":[],"garbage_collect_requested":[]}
45652025-05-30T21:35:46.003ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45662025-05-30T21:35:46.005ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45672025-05-30T21:35:46.006ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45682025-05-30T21:35:46.006ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
45692025-05-30T21:35:46.006ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
45702025-05-30T21:35:46.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
45712025-05-30T21:35:46.007ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
45722025-05-30T21:35:46.007ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 65.455121ms iteration = 20 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
45732025-05-30T21:35:46.007ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
45742025-05-30T21:35:46.008ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 66.082339ms iteration = 20 status = {"errors":[],"requests_created_ok":[]}
45752025-05-30T21:35:46.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
45762025-05-30T21:35:46.008ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45772025-05-30T21:35:46.008ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45782025-05-30T21:35:46.008ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
45792025-05-30T21:35:46.008ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
45802025-05-30T21:35:46.008ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45812025-05-30T21:35:46.009ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
45822025-05-30T21:35:46.009ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 67.095241ms iteration = 20 status = {"errors":[],"finish_invoked_ok":[]}
45832025-05-30T21:35:46.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
45842025-05-30T21:35:46.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45852025-05-30T21:35:46.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
45862025-05-30T21:35:46.010ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
45872025-05-30T21:35:46.010ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
45882025-05-30T21:35:46.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
45892025-05-30T21:35:46.011ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
45902025-05-30T21:35:46.011ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
45912025-05-30T21:35:46.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
45922025-05-30T21:35:46.012ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
45932025-05-30T21:35:46.012ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
45942025-05-30T21:35:46.012ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45952025-05-30T21:35:46.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45962025-05-30T21:35:46.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
45972025-05-30T21:35:46.012ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
45982025-05-30T21:35:46.012ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
45992025-05-30T21:35:46.012ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
46002025-05-30T21:35:46.013ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
46012025-05-30T21:35:46.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
46022025-05-30T21:35:46.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
46032025-05-30T21:35:46.015ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46042025-05-30T21:35:46.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46052025-05-30T21:35:46.016ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46062025-05-30T21:35:46.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46072025-05-30T21:35:46.016ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1680, tv_nsec: 953453957 } })
46082025-05-30T21:35:46.017ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
46092025-05-30T21:35:46.017ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 75.641899ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
46102025-05-30T21:35:46.046ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46112025-05-30T21:35:46.047ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 105.889626ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
46122025-05-30T21:35:46.049ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
46132025-05-30T21:35:46.049ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
46142025-05-30T21:35:46.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
46152025-05-30T21:35:46.055ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46162025-05-30T21:35:46.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46172025-05-30T21:35:46.056ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 115.262644ms iteration = 58 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
46182025-05-30T21:35:46.058ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 116.632159ms iteration = 20 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
46192025-05-30T21:35:46.058ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46202025-05-30T21:35:46.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46212025-05-30T21:35:46.059ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46222025-05-30T21:35:46.060ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46232025-05-30T21:35:46.060ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
46242025-05-30T21:35:46.060ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
46252025-05-30T21:35:46.060ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 119.138819ms iteration = 20 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
46262025-05-30T21:35:46.062ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
46272025-05-30T21:35:46.062ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
46282025-05-30T21:35:46.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46292025-05-30T21:35:46.063ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46302025-05-30T21:35:46.064ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
46312025-05-30T21:35:46.064ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46322025-05-30T21:35:46.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46332025-05-30T21:35:46.065ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
46342025-05-30T21:35:46.065ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46352025-05-30T21:35:46.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46362025-05-30T21:35:46.067ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 125.33505ms iteration = 20 status = 0
46372025-05-30T21:35:46.067ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46382025-05-30T21:35:46.071ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
46392025-05-30T21:35:46.071ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46402025-05-30T21:35:46.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46412025-05-30T21:35:46.083ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
46422025-05-30T21:35:46.083ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 141.264463ms iteration = 58 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
46432025-05-30T21:35:46.098ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
46442025-05-30T21:35:46.098ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 156.494778ms iteration = 20 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
46452025-05-30T21:35:46.114ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
46462025-05-30T21:35:46.114ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
46472025-05-30T21:35:46.115ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
46482025-05-30T21:35:46.115ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
46492025-05-30T21:35:46.115ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "51cff8ca-fbfe-4b91-ba95-665fd39bc629", "content-length": "26", "date": "Fri, 30 May 2025 21:35:46 GMT"} })
46502025-05-30T21:35:46.115ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
46512025-05-30T21:35:46.115ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "73f3f511-61f4-46bd-abaf-139ce4087c4b", "content-length": "26", "date": "Fri, 30 May 2025 21:35:46 GMT"} })
46522025-05-30T21:35:46.115ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
46532025-05-30T21:35:46.117ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
46542025-05-30T21:35:46.117ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
46552025-05-30T21:35:46.118ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2178ec5-9b8e-4252-b77a-401539592f2f", "content-length": "26", "date": "Fri, 30 May 2025 21:35:46 GMT"} })
46562025-05-30T21:35:46.118ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
46572025-05-30T21:35:46.118ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
46582025-05-30T21:35:46.118ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
46592025-05-30T21:35:46.118ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
46602025-05-30T21:35:46.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46612025-05-30T21:35:46.119ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
46622025-05-30T21:35:46.174ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
46632025-05-30T21:35:46.174ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
46642025-05-30T21:35:46.175ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "a34153b6-1a58-4a5a-9f1d-d83cd00b919c", "content-length": "372", "date": "Fri, 30 May 2025 21:35:45 GMT"} })
46652025-05-30T21:35:46.175ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46662025-05-30T21:35:46.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46672025-05-30T21:35:46.190ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
46682025-05-30T21:35:46.190ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
46692025-05-30T21:35:46.190ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "45e7b83d-1c8f-4251-9e11-ce4481c15d8c", "date": "Fri, 30 May 2025 21:35:45 GMT"} })
46702025-05-30T21:35:46.190ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46712025-05-30T21:35:46.190ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46722025-05-30T21:35:46.192ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 250.141873ms iteration = 21 status = {"physical_disks_added":0}
46732025-05-30T21:35:46.193ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46742025-05-30T21:35:46.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46752025-05-30T21:35:46.196ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46762025-05-30T21:35:46.197ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46772025-05-30T21:35:46.200ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
46782025-05-30T21:35:46.200ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
46792025-05-30T21:35:46.200ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "261e02fa-0d3d-43a0-9eb1-000270b33239", "date": "Fri, 30 May 2025 21:35:45 GMT"} })
46802025-05-30T21:35:46.202ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 260.656364ms iteration = 20 status = {}
46812025-05-30T21:35:46.226ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
46822025-05-30T21:35:46.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46832025-05-30T21:35:46.228ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46842025-05-30T21:35:46.230ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
46852025-05-30T21:35:46.230ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
46862025-05-30T21:35:46.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46872025-05-30T21:35:46.231ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46882025-05-30T21:35:46.233ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
46892025-05-30T21:35:46.233ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
46902025-05-30T21:35:46.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46912025-05-30T21:35:46.306ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 364.326915ms iteration = 20 status = {}
46922025-05-30T21:35:46.323ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
46932025-05-30T21:35:46.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
46942025-05-30T21:35:46.330ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 388.938867ms iteration = 20 status = null
46952025-05-30T21:35:46.380ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
46962025-05-30T21:35:46.380ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ce00c55-8d06-4700-a11c-e1225ff5540e", "content-length": "691", "date": "Fri, 30 May 2025 21:35:45 GMT"} })
46972025-05-30T21:35:46.380ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
46982025-05-30T21:35:46.380ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
46992025-05-30T21:35:46.380ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
47002025-05-30T21:35:46.381ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 439.890102ms iteration = 20 status = {}
47012025-05-30T21:35:46.381ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47022025-05-30T21:35:46.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47032025-05-30T21:35:46.384ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
47042025-05-30T21:35:46.384ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47052025-05-30T21:35:46.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47062025-05-30T21:35:46.397ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
47072025-05-30T21:35:46.397ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
47082025-05-30T21:35:46.397ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
47092025-05-30T21:35:46.397ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
47102025-05-30T21:35:46.397ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47112025-05-30T21:35:46.397ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47122025-05-30T21:35:46.397ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47132025-05-30T21:35:46.397ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
47142025-05-30T21:35:46.397ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47152025-05-30T21:35:46.397ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
47162025-05-30T21:35:46.397ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
47172025-05-30T21:35:46.397ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47182025-05-30T21:35:46.398ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47192025-05-30T21:35:46.400ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
47202025-05-30T21:35:46.400ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
47212025-05-30T21:35:46.502ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "89ea0659-4a23-4587-8725-dfdd2eeac12b", "date": "Fri, 30 May 2025 21:35:45 GMT"} })
47222025-05-30T21:35:46.502ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
47232025-05-30T21:35:46.502ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
47242025-05-30T21:35:46.506ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "179ac52d-e495-4dae-a4b1-011a033320b4", "content-length": "682", "date": "Fri, 30 May 2025 21:35:46 GMT"} })
47252025-05-30T21:35:46.506ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47262025-05-30T21:35:46.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47272025-05-30T21:35:46.510ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47282025-05-30T21:35:46.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47292025-05-30T21:35:46.513ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47302025-05-30T21:35:46.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47312025-05-30T21:35:46.516ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47322025-05-30T21:35:46.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47332025-05-30T21:35:46.527ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 585.453014ms iteration = 20 status = {}
47342025-05-30T21:35:48.776ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 316 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 591616f0-6b90-48d7-8685-908c2db2e9d5 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
47352025-05-30T21:35:55.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 59 reason = Timeout
47362025-05-30T21:35:55.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 59 reason = Timeout
47372025-05-30T21:35:55.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47382025-05-30T21:35:55.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47392025-05-30T21:35:55.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47402025-05-30T21:35:55.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47412025-05-30T21:35:55.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47422025-05-30T21:35:55.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
47432025-05-30T21:35:55.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47442025-05-30T21:35:55.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47452025-05-30T21:35:55.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47462025-05-30T21:35:55.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47472025-05-30T21:35:55.975ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47482025-05-30T21:35:55.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47492025-05-30T21:35:55.977ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 35.037492ms iteration = 59 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
47502025-05-30T21:35:55.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47512025-05-30T21:35:55.984ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
47522025-05-30T21:35:55.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47532025-05-30T21:35:55.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47542025-05-30T21:35:55.994ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
47552025-05-30T21:35:55.994ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 52.016518ms iteration = 59 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
47562025-05-30T21:35:58.777ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 368 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 0fdd2a93-c68c-407f-bdc3-10cc3cf816c1 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
47572025-05-30T21:36:05.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 60 reason = Timeout
47582025-05-30T21:36:05.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 60 reason = Timeout
47592025-05-30T21:36:05.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47602025-05-30T21:36:05.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47612025-05-30T21:36:05.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47622025-05-30T21:36:05.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47632025-05-30T21:36:05.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47642025-05-30T21:36:05.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47652025-05-30T21:36:05.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47662025-05-30T21:36:05.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
47672025-05-30T21:36:05.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47682025-05-30T21:36:05.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47692025-05-30T21:36:05.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47702025-05-30T21:36:05.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47712025-05-30T21:36:05.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47722025-05-30T21:36:05.952ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
47732025-05-30T21:36:05.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47742025-05-30T21:36:05.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47752025-05-30T21:36:05.962ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
47762025-05-30T21:36:05.962ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.010681ms iteration = 60 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
47772025-05-30T21:36:05.980ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 38.248239ms iteration = 60 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
47782025-05-30T21:36:08.777ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 346 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = e6706551-f9d2-4a64-a07c-0a6b05a57977 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
47792025-05-30T21:36:15.940ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
47802025-05-30T21:36:15.940ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
47812025-05-30T21:36:15.940ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
47822025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
47832025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 61 reason = Timeout
47842025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47852025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 11 reason = Timeout
47862025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
47872025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 11 reason = Timeout
47882025-05-30T21:36:15.941ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-30 21:26:15.941377570 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
47892025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 3 reason = Timeout
47902025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
47912025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 12 reason = Timeout
47922025-05-30T21:36:15.941ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e
47932025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
47942025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.44\u{b5}s iteration = 12 status = {"enabled":false,"target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e"}
47952025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
47962025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 3 reason = Timeout
47972025-05-30T21:36:15.941ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
47982025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
47992025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
48002025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Timeout
48012025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48022025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 2 reason = Timeout
48032025-05-30T21:36:15.941ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
48042025-05-30T21:36:15.941ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
48052025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 4 reason = Timeout
48062025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 11 reason = Timeout
48072025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 11 reason = Timeout
48082025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
48092025-05-30T21:36:15.941ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
48102025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
48112025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
48122025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 21 reason = Timeout
48132025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
48142025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
48152025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 11 reason = Timeout
48162025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
48172025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48182025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 11 reason = Timeout
48192025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 21 reason = Timeout
48202025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
48212025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48222025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 11 reason = Timeout
48232025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
48242025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
48252025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
48262025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
48272025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 21 reason = Timeout
48282025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
48292025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 21 reason = Timeout
48302025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
48312025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 21 reason = Timeout
48322025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
48332025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 21 reason = Timeout
48342025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
48352025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
48362025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
48372025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
48382025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
48392025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 21 reason = Timeout
48402025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 21 reason = Timeout
48412025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 22 reason = Timeout
48422025-05-30T21:36:15.941ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
48432025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48442025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
48452025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 21 reason = Timeout
48462025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
48472025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 21 reason = Timeout
48482025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
48492025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 21 reason = Timeout
48502025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
48512025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 21 reason = Timeout
48522025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
48532025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 21 reason = Timeout
48542025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
48552025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 21 reason = Timeout
48562025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 21 reason = Timeout
48572025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
48582025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
48592025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 21 reason = Timeout
48602025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48612025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 61 reason = Timeout
48622025-05-30T21:36:15.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48632025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
48642025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
48652025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48662025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
48672025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48682025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
48692025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
48702025-05-30T21:36:15.942ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
48712025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
48722025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
48732025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
48742025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
48752025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
48762025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
48772025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
48782025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
48792025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
48802025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
48812025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
48822025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
48832025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
48842025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48852025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
48862025-05-30T21:36:15.942ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0413f6b4-86a1-42a1-919a-e8f42ff9f638.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("acb1452d-0a02-4b36-a462-c8ce2f96a488.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:2::1))) }], valid_until: Instant { tv_sec: 1710, tv_nsec: 878881472 } })
48872025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
48882025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48892025-05-30T21:36:15.942ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2a161087-f464-41f8-b020-0f9e83fd5351.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1710, tv_nsec: 878892172 } })
48902025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
48912025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
48922025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48932025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
48942025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
48952025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
48962025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
48972025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
48982025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
48992025-05-30T21:36:15.942ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.473706ms iteration = 11 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
49002025-05-30T21:36:15.943ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } dns_group = internal
49012025-05-30T21:36:15.943ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.595258ms iteration = 11 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353"]}
49022025-05-30T21:36:15.944ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49032025-05-30T21:36:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49042025-05-30T21:36:15.946ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.985397ms iteration = 21 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
49052025-05-30T21:36:15.947ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-30 21:26:15.941377570 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
49062025-05-30T21:36:15.947ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 6.385913ms iteration = 11 status = {"expiration":"2025-05-30T21:26:15.941377570Z","pruned":[]}
49072025-05-30T21:36:15.949ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49082025-05-30T21:36:15.950ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49092025-05-30T21:36:15.952ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
49102025-05-30T21:36:15.952ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
49112025-05-30T21:36:15.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
49122025-05-30T21:36:15.953ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49132025-05-30T21:36:15.953ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49142025-05-30T21:36:15.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49152025-05-30T21:36:15.954ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
49162025-05-30T21:36:15.954ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49172025-05-30T21:36:15.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49182025-05-30T21:36:15.956ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49192025-05-30T21:36:15.956ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49202025-05-30T21:36:15.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49212025-05-30T21:36:15.956ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49222025-05-30T21:36:15.958ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:202 num_receivers = 0
49232025-05-30T21:36:15.958ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 16.836944ms iteration = 11 status = {"by_rx":{},"error":null}
49242025-05-30T21:36:15.959ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49252025-05-30T21:36:15.959ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
49262025-05-30T21:36:15.959ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
49272025-05-30T21:36:15.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
49282025-05-30T21:36:15.961ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
49292025-05-30T21:36:15.961ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
49302025-05-30T21:36:15.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
49312025-05-30T21:36:15.963ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49322025-05-30T21:36:15.964ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
49332025-05-30T21:36:15.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
49342025-05-30T21:36:15.965ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49352025-05-30T21:36:15.965ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 24.226528ms iteration = 11 status = {"errors":[],"found_port_ok":[]}
49362025-05-30T21:36:15.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
49372025-05-30T21:36:15.967ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
49382025-05-30T21:36:15.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
49392025-05-30T21:36:15.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
49402025-05-30T21:36:15.968ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
49412025-05-30T21:36:15.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
49422025-05-30T21:36:15.969ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1710, tv_nsec: 905398557 } })
49432025-05-30T21:36:15.969ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
49442025-05-30T21:36:15.973ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49452025-05-30T21:36:15.973ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49462025-05-30T21:36:15.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
49472025-05-30T21:36:15.976ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49482025-05-30T21:36:15.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49492025-05-30T21:36:15.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
49502025-05-30T21:36:15.976ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
49512025-05-30T21:36:15.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
49522025-05-30T21:36:15.978ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49532025-05-30T21:36:15.979ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1710, tv_nsec: 915463862 } })
49542025-05-30T21:36:15.979ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
49552025-05-30T21:36:15.983ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
49562025-05-30T21:36:15.988ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
49572025-05-30T21:36:15.988ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49582025-05-30T21:36:15.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
49592025-05-30T21:36:15.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49602025-05-30T21:36:15.994ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49612025-05-30T21:36:15.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49622025-05-30T21:36:16.001ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(9b585029-489f-4bf8-a810-ff0d8e4f15bd, 0), (a75b8315-ce25-40db-961f-8951af28f0d7, 0), (0165edca-c075-4a18-9f51-5e948173b5ec, 0)]
49632025-05-30T21:36:16.002ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1710, tv_nsec: 938697959 } })
49642025-05-30T21:36:16.003ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
49652025-05-30T21:36:16.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
49662025-05-30T21:36:16.004ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 62.858472ms iteration = 21 status = {"errors":[],"garbage_collect_requested":[]}
49672025-05-30T21:36:16.007ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49682025-05-30T21:36:16.007ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
49692025-05-30T21:36:16.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
49702025-05-30T21:36:16.009ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
49712025-05-30T21:36:16.009ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
49722025-05-30T21:36:16.009ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
49732025-05-30T21:36:16.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
49742025-05-30T21:36:16.010ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
49752025-05-30T21:36:16.010ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
49762025-05-30T21:36:16.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
49772025-05-30T21:36:16.013ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49782025-05-30T21:36:16.014ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
49792025-05-30T21:36:16.014ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
49802025-05-30T21:36:16.015ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 73.353793ms iteration = 11 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
49812025-05-30T21:36:16.017ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
49822025-05-30T21:36:16.019ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
49832025-05-30T21:36:16.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
49842025-05-30T21:36:16.021ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49852025-05-30T21:36:16.022ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1710, tv_nsec: 958560297 } })
49862025-05-30T21:36:16.022ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
49872025-05-30T21:36:16.022ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
49882025-05-30T21:36:16.022ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 81.240713ms iteration = 21 status = {"errors":[],"requests_created_ok":[]}
49892025-05-30T21:36:16.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
49902025-05-30T21:36:16.023ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
49912025-05-30T21:36:16.023ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49922025-05-30T21:36:16.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49932025-05-30T21:36:16.026ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
49942025-05-30T21:36:16.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
49952025-05-30T21:36:16.027ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49962025-05-30T21:36:16.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49972025-05-30T21:36:16.028ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
49982025-05-30T21:36:16.028ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 87.002759ms iteration = 11 status = {"dispatched":[],"errors":[],"glob_version":"144.0.0","globs_reprocessed":{},"no_receivers":[]}
49992025-05-30T21:36:16.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
50002025-05-30T21:36:16.029ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50012025-05-30T21:36:16.030ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 88.253843ms iteration = 21 status = {"errors":[],"finish_invoked_ok":[]}
50022025-05-30T21:36:16.030ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
50032025-05-30T21:36:16.030ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 89.022173ms iteration = 2 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
50042025-05-30T21:36:16.032ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50052025-05-30T21:36:16.032ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
50062025-05-30T21:36:16.032ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
50072025-05-30T21:36:16.032ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
50082025-05-30T21:36:16.032ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50092025-05-30T21:36:16.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
50102025-05-30T21:36:16.034ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 92.915798ms iteration = 61 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
50112025-05-30T21:36:16.034ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50122025-05-30T21:36:16.035ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
50132025-05-30T21:36:16.035ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
50142025-05-30T21:36:16.035ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
50152025-05-30T21:36:16.035ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50162025-05-30T21:36:16.035ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
50172025-05-30T21:36:16.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
50182025-05-30T21:36:16.035ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
50192025-05-30T21:36:16.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50202025-05-30T21:36:16.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
50212025-05-30T21:36:16.039ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
50222025-05-30T21:36:16.039ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50232025-05-30T21:36:16.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50242025-05-30T21:36:16.040ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
50252025-05-30T21:36:16.040ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
50262025-05-30T21:36:16.040ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
50272025-05-30T21:36:16.040ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
50282025-05-30T21:36:16.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
50292025-05-30T21:36:16.043ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50302025-05-30T21:36:16.043ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
50312025-05-30T21:36:16.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50322025-05-30T21:36:16.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
50332025-05-30T21:36:16.044ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50342025-05-30T21:36:16.045ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50352025-05-30T21:36:16.046ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50362025-05-30T21:36:16.047ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
50372025-05-30T21:36:16.047ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
50382025-05-30T21:36:16.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
50392025-05-30T21:36:16.050ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
50402025-05-30T21:36:16.050ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50412025-05-30T21:36:16.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50422025-05-30T21:36:16.050ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
50432025-05-30T21:36:16.050ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
50442025-05-30T21:36:16.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
50452025-05-30T21:36:16.060ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
50462025-05-30T21:36:16.080ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50472025-05-30T21:36:16.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50482025-05-30T21:36:16.095ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50492025-05-30T21:36:16.096ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50502025-05-30T21:36:16.098ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
50512025-05-30T21:36:16.098ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
50522025-05-30T21:36:16.098ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 157.145315ms iteration = 11 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
50532025-05-30T21:36:16.099ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50542025-05-30T21:36:16.099ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 158.113646ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
50552025-05-30T21:36:16.101ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50562025-05-30T21:36:16.101ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
50572025-05-30T21:36:16.101ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
50582025-05-30T21:36:16.101ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
50592025-05-30T21:36:16.101ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
50602025-05-30T21:36:16.101ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
50612025-05-30T21:36:16.101ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 160.076628ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
50622025-05-30T21:36:16.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
50632025-05-30T21:36:16.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
50642025-05-30T21:36:16.103ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
50652025-05-30T21:36:16.103ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50662025-05-30T21:36:16.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50672025-05-30T21:36:16.104ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
50682025-05-30T21:36:16.104ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
50692025-05-30T21:36:16.104ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 163.200134ms iteration = 21 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
50702025-05-30T21:36:16.106ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50712025-05-30T21:36:16.106ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
50722025-05-30T21:36:16.106ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
50732025-05-30T21:36:16.106ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50742025-05-30T21:36:16.107ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
50752025-05-30T21:36:16.107ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 165.44961ms iteration = 21 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
50762025-05-30T21:36:16.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50772025-05-30T21:36:16.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
50782025-05-30T21:36:16.107ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50792025-05-30T21:36:16.108ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
50802025-05-30T21:36:16.108ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 166.838376ms iteration = 61 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
50812025-05-30T21:36:16.109ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
50822025-05-30T21:36:16.109ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
50832025-05-30T21:36:16.109ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
50842025-05-30T21:36:16.112ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50852025-05-30T21:36:16.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50862025-05-30T21:36:16.112ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
50872025-05-30T21:36:16.112ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
50882025-05-30T21:36:16.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
50892025-05-30T21:36:16.114ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 172.872186ms iteration = 21 status = 0
50902025-05-30T21:36:16.123ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50912025-05-30T21:36:16.123ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50922025-05-30T21:36:16.123ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50932025-05-30T21:36:16.123ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
50942025-05-30T21:36:16.123ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 181.658097ms iteration = 21 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
50952025-05-30T21:36:16.123ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
50962025-05-30T21:36:16.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50972025-05-30T21:36:16.124ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
50982025-05-30T21:36:16.125ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50992025-05-30T21:36:16.126ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
51002025-05-30T21:36:16.126ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
51012025-05-30T21:36:16.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
51022025-05-30T21:36:16.126ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
51032025-05-30T21:36:16.126ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51042025-05-30T21:36:16.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51052025-05-30T21:36:16.128ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
51062025-05-30T21:36:16.131ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
51072025-05-30T21:36:16.131ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51082025-05-30T21:36:16.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51092025-05-30T21:36:16.134ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
51102025-05-30T21:36:16.134ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
51112025-05-30T21:36:16.134ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "68f90c95-9130-47cc-bbc8-9d381fb3091e", "content-length": "26", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
51122025-05-30T21:36:16.134ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
51132025-05-30T21:36:16.137ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
51142025-05-30T21:36:16.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51152025-05-30T21:36:16.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51162025-05-30T21:36:16.141ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
51172025-05-30T21:36:16.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
51182025-05-30T21:36:16.141ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51192025-05-30T21:36:16.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51202025-05-30T21:36:16.142ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51212025-05-30T21:36:16.145ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
51222025-05-30T21:36:16.145ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51232025-05-30T21:36:16.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51242025-05-30T21:36:16.148ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51252025-05-30T21:36:16.148ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51262025-05-30T21:36:16.148ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51272025-05-30T21:36:16.150ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51282025-05-30T21:36:16.151ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
51292025-05-30T21:36:16.151ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51302025-05-30T21:36:16.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51312025-05-30T21:36:16.155ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
51322025-05-30T21:36:16.155ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
51332025-05-30T21:36:16.155ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51342025-05-30T21:36:16.155ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 214.197281ms iteration = 4 status = {"blueprint_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","inventory_collection_id":"0165edca-c075-4a18-9f51-5e948173b5ec","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
51352025-05-30T21:36:16.155ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51362025-05-30T21:36:16.162ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51372025-05-30T21:36:16.162ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51382025-05-30T21:36:16.166ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51392025-05-30T21:36:16.166ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51402025-05-30T21:36:16.168ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
51412025-05-30T21:36:16.168ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
51422025-05-30T21:36:16.168ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
51432025-05-30T21:36:16.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
51442025-05-30T21:36:16.169ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51452025-05-30T21:36:16.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51462025-05-30T21:36:16.171ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51472025-05-30T21:36:16.172ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51482025-05-30T21:36:16.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51492025-05-30T21:36:16.175ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51502025-05-30T21:36:16.175ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
51512025-05-30T21:36:16.175ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
51522025-05-30T21:36:16.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
51532025-05-30T21:36:16.176ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51542025-05-30T21:36:16.181ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51552025-05-30T21:36:16.181ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51562025-05-30T21:36:16.181ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
51572025-05-30T21:36:16.181ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
51582025-05-30T21:36:16.181ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-30 21:26:17.226115 UTC dns_group = internal generation = 1
51592025-05-30T21:36:16.181ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 240.872727ms iteration = 11 status = {"generation":1}
51602025-05-30T21:36:16.183ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51612025-05-30T21:36:16.189ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
51622025-05-30T21:36:16.189ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51632025-05-30T21:36:16.190ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51642025-05-30T21:36:16.190ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
51652025-05-30T21:36:16.190ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 0 version = 2
51662025-05-30T21:36:16.190ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
51672025-05-30T21:36:16.190ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
51682025-05-30T21:36:16.192ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51692025-05-30T21:36:16.192ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51702025-05-30T21:36:16.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51712025-05-30T21:36:16.195ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
51722025-05-30T21:36:16.195ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51732025-05-30T21:36:16.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51742025-05-30T21:36:16.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
51752025-05-30T21:36:16.196ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
51762025-05-30T21:36:16.196ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
51772025-05-30T21:36:16.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
51782025-05-30T21:36:16.205ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 1 version = 2
51792025-05-30T21:36:16.205ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
51802025-05-30T21:36:16.205ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-30 21:26:18.957109 UTC dns_group = external generation = 2
51812025-05-30T21:36:16.205ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 264.163294ms iteration = 11 status = {"generation":2}
51822025-05-30T21:36:16.209ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51832025-05-30T21:36:16.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51842025-05-30T21:36:16.221ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51852025-05-30T21:36:16.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51862025-05-30T21:36:16.223ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-30T21:26:13.986396Z, time_modified: 2025-05-30T21:26:13.986396Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }) }
51872025-05-30T21:36:16.223ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 282.258581ms iteration = 11 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]}},"default_endpoint":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
51882025-05-30T21:36:16.225ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 283.82147ms iteration = 21 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
51892025-05-30T21:36:16.226ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51902025-05-30T21:36:16.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51912025-05-30T21:36:16.229ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
51922025-05-30T21:36:16.229ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
51932025-05-30T21:36:16.229ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0820a2d6-d289-4413-9bf4-92e5a0954393", "content-length": "26", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
51942025-05-30T21:36:16.229ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
51952025-05-30T21:36:16.230ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
51962025-05-30T21:36:16.230ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
51972025-05-30T21:36:16.231ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb731a1e-3167-4efc-be60-054e3ea15c71", "content-length": "26", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
51982025-05-30T21:36:16.231ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
51992025-05-30T21:36:16.236ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
52002025-05-30T21:36:16.243ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
52012025-05-30T21:36:16.258ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
52022025-05-30T21:36:16.258ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
52032025-05-30T21:36:16.258ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
52042025-05-30T21:36:16.259ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a0326e6-84a2-415d-86ea-c35674e28ff0", "content-length": "476", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52052025-05-30T21:36:16.259ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
52062025-05-30T21:36:16.259ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b66e1b2b-c67f-463e-b77f-1ce0d35c23aa", "content-length": "734", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52072025-05-30T21:36:16.259ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
52082025-05-30T21:36:16.260ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6f90509-be7f-4d84-aec6-7d11b0a58961", "content-length": "111", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52092025-05-30T21:36:16.260ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
52102025-05-30T21:36:16.260ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82d5a66b-52f1-4eb3-b2de-5a977170a989", "content-length": "111", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52112025-05-30T21:36:16.260ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
52122025-05-30T21:36:16.260ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be82f59d-e780-44e5-8d7e-3181a73983ca", "content-length": "170", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52132025-05-30T21:36:16.261ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
52142025-05-30T21:36:16.261ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbc16dfb-909f-4dcb-a354-fa58481e116b", "content-length": "170", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52152025-05-30T21:36:16.261ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
52162025-05-30T21:36:16.261ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "388b8d36-ef13-4ae3-86dd-be234a978726", "content-length": "179", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52172025-05-30T21:36:16.261ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
52182025-05-30T21:36:16.262ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0b628c7-e753-4a0e-9310-803ee7b645f3", "content-length": "179", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52192025-05-30T21:36:16.262ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
52202025-05-30T21:36:16.262ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "189bd220-0f72-4491-bcfa-104c1ede7daa", "content-length": "702", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52212025-05-30T21:36:16.262ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
52222025-05-30T21:36:16.262ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ed3ce66-7335-49d0-9729-8c9ba50e79c0", "content-length": "727", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52232025-05-30T21:36:16.262ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
52242025-05-30T21:36:16.262ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe70f64a-df88-48eb-8522-a8142f4aee1a", "content-length": "729", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52252025-05-30T21:36:16.262ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
52262025-05-30T21:36:16.262ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccf7d460-a4ca-413e-8c05-89ea9eb5a4ff", "content-length": "728", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52272025-05-30T21:36:16.262ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
52282025-05-30T21:36:16.263ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14292d68-8914-412c-9f1b-0b698a9aad74", "content-length": "722", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52292025-05-30T21:36:16.263ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
52302025-05-30T21:36:16.263ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c67ed0b5-5bc5-4bbb-a4f6-0d189aa670ac", "content-length": "109", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52312025-05-30T21:36:16.263ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
52322025-05-30T21:36:16.263ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "684f40ea-71ab-4b6a-ab97-c96f6d46ca39", "content-length": "109", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52332025-05-30T21:36:16.263ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
52342025-05-30T21:36:16.264ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5f50699-0126-4859-8636-6f24982518fd", "content-length": "169", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52352025-05-30T21:36:16.264ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
52362025-05-30T21:36:16.264ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dab75d85-5517-4c7b-a314-fa10397da51a", "content-length": "169", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52372025-05-30T21:36:16.264ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
52382025-05-30T21:36:16.264ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cb22a59-4587-4ebb-ae45-d076c0a7c026", "content-length": "178", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52392025-05-30T21:36:16.264ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
52402025-05-30T21:36:16.265ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "288b2969-7482-46cf-a3ef-06e1093233d6", "content-length": "178", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52412025-05-30T21:36:16.265ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
52422025-05-30T21:36:16.265ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b367d6e8-03f5-4aed-9e3d-c9fecbc590b6", "content-length": "702", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52432025-05-30T21:36:16.265ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
52442025-05-30T21:36:16.265ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e69f70a-7bcf-4245-bf94-c746362b3632", "content-length": "727", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52452025-05-30T21:36:16.265ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
52462025-05-30T21:36:16.265ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9632f089-5c92-45f4-b27a-58882ab0ad47", "content-length": "729", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52472025-05-30T21:36:16.266ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
52482025-05-30T21:36:16.266ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3379972a-f3bd-49f6-ada0-f0eeea86db54", "content-length": "728", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
52492025-05-30T21:36:16.266ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52502025-05-30T21:36:16.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52512025-05-30T21:36:16.271ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
52522025-05-30T21:36:16.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
52532025-05-30T21:36:16.292ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
52542025-05-30T21:36:16.292ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "e85d8ec2-9f43-4a6d-9c3b-a14cd13ea9c2", "date": "Fri, 30 May 2025 21:36:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
52552025-05-30T21:36:16.292ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
52562025-05-30T21:36:16.294ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "08ae3839-08a0-43c6-b43a-431e788e198b", "date": "Fri, 30 May 2025 21:36:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
52572025-05-30T21:36:16.294ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
52582025-05-30T21:36:16.296ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
52592025-05-30T21:36:16.296ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 354.972508ms iteration = 13 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
52602025-05-30T21:36:16.307ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
52612025-05-30T21:36:16.307ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 366.127245ms iteration = 11 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
52622025-05-30T21:36:16.308ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 366.875254ms iteration = 22 status = {"physical_disks_added":0}
52632025-05-30T21:36:16.344ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
52642025-05-30T21:36:16.344ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
52652025-05-30T21:36:16.385ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
52662025-05-30T21:36:16.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
52672025-05-30T21:36:16.390ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
52682025-05-30T21:36:16.390ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
52692025-05-30T21:36:16.390ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cd908bb-558b-457c-b109-f9775875b52e", "content-length": "372", "date": "Fri, 30 May 2025 21:36:15 GMT"} })
52702025-05-30T21:36:16.390ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52712025-05-30T21:36:16.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52722025-05-30T21:36:16.393ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52732025-05-30T21:36:16.394ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 452.507018ms iteration = 21 status = null
52742025-05-30T21:36:16.394ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52752025-05-30T21:36:16.395ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
52762025-05-30T21:36:16.395ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
52772025-05-30T21:36:16.395ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
52782025-05-30T21:36:16.397ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
52792025-05-30T21:36:16.397ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
52802025-05-30T21:36:16.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
52812025-05-30T21:36:16.399ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
52822025-05-30T21:36:16.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
52832025-05-30T21:36:16.424ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 482.835416ms iteration = 21 status = {}
52842025-05-30T21:36:16.434ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52852025-05-30T21:36:16.434ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
52862025-05-30T21:36:16.434ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52872025-05-30T21:36:16.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
52882025-05-30T21:36:16.437ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
52892025-05-30T21:36:16.437ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
52902025-05-30T21:36:16.437ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
52912025-05-30T21:36:16.437ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52922025-05-30T21:36:16.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
52932025-05-30T21:36:16.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52942025-05-30T21:36:16.438ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52952025-05-30T21:36:16.442ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
52962025-05-30T21:36:16.442ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
52972025-05-30T21:36:16.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
52982025-05-30T21:36:16.443ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52992025-05-30T21:36:16.443ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53002025-05-30T21:36:16.444ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
53012025-05-30T21:36:16.444ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53022025-05-30T21:36:16.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53032025-05-30T21:36:16.445ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
53042025-05-30T21:36:16.445ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53052025-05-30T21:36:16.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53062025-05-30T21:36:16.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
53072025-05-30T21:36:16.446ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
53082025-05-30T21:36:16.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
53092025-05-30T21:36:16.447ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/artifacts-config
53102025-05-30T21:36:16.448ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts-config", status: 204, headers: {"x-request-id": "250465ef-3be5-4c60-8a19-bf729b0b2491", "date": "Fri, 30 May 2025 21:36:15 GMT"} })
53112025-05-30T21:36:16.448ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
53122025-05-30T21:36:16.448ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "f33a760a-f7c9-4873-aacb-c79e77d14ffb", "content-length": "26", "date": "Fri, 30 May 2025 21:36:15 GMT"} })
53132025-05-30T21:36:16.448ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:673 sled = http://[fd00:1122:3344:101::1]:12345
53142025-05-30T21:36:16.449ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
53152025-05-30T21:36:16.449ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
53162025-05-30T21:36:16.449ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "287870cc-dffb-4606-b630-0d7e63533180", "date": "Fri, 30 May 2025 21:36:15 GMT"} })
53172025-05-30T21:36:16.449ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53182025-05-30T21:36:16.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53192025-05-30T21:36:16.450ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 509.369181ms iteration = 3 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":2,"put_config_err":0,"put_config_ok":2,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"5d926227-6a52-4938-b74d-21ba39bce2f9","time":"2025-05-30T21:31:16.555348283Z"},{"operation":"list","target_sled":"5d926227-6a52-4938-b74d-21ba39bce2f9","time":"2025-05-30T21:31:16.558295177Z"},{"generation":1,"operation":"put_config","target_sled":"5d926227-6a52-4938-b74d-21ba39bce2f9","time":"2025-05-30T21:36:16.448455654Z"},{"operation":"list","target_sled":"5d926227-6a52-4938-b74d-21ba39bce2f9","time":"2025-05-30T21:36:16.448677367Z"}]}
53202025-05-30T21:36:16.452ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53212025-05-30T21:36:16.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53222025-05-30T21:36:16.483ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53232025-05-30T21:36:16.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53242025-05-30T21:36:16.555ZDEBGnexus (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[fd00:1122:3344:101::1]:12345
53252025-05-30T21:36:16.555ZDEBGnexus (BackgroundTasks): client request SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/inventory
53262025-05-30T21:36:16.558ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53272025-05-30T21:36:16.559ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53282025-05-30T21:36:16.559ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53292025-05-30T21:36:16.563ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
53302025-05-30T21:36:16.563ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
53312025-05-30T21:36:16.563ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53322025-05-30T21:36:16.563ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
53332025-05-30T21:36:16.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53342025-05-30T21:36:16.563ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "88f1b107-5fce-4813-8267-5ee65b6acfdf", "date": "Fri, 30 May 2025 21:36:15 GMT"} })
53352025-05-30T21:36:16.565ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 623.835565ms iteration = 21 status = {}
53362025-05-30T21:36:16.569ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53372025-05-30T21:36:16.569ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53382025-05-30T21:36:16.571ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53392025-05-30T21:36:16.572ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53402025-05-30T21:36:16.575ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53412025-05-30T21:36:16.576ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53422025-05-30T21:36:16.580ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53432025-05-30T21:36:16.581ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53442025-05-30T21:36:16.594ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53452025-05-30T21:36:16.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53462025-05-30T21:36:16.596ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53472025-05-30T21:36:16.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53482025-05-30T21:36:16.598ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53492025-05-30T21:36:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53502025-05-30T21:36:16.624ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
53512025-05-30T21:36:16.624ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdcf5183-b9bc-426b-b738-c147e96e2dcf", "content-length": "691", "date": "Fri, 30 May 2025 21:36:15 GMT"} })
53522025-05-30T21:36:16.624ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
53532025-05-30T21:36:16.624ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
53542025-05-30T21:36:16.624ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
53552025-05-30T21:36:16.625ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 684.085297ms iteration = 21 status = {}
53562025-05-30T21:36:16.626ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53572025-05-30T21:36:16.630ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
53582025-05-30T21:36:16.630ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53592025-05-30T21:36:16.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53602025-05-30T21:36:16.632ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53612025-05-30T21:36:16.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53622025-05-30T21:36:16.633ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53632025-05-30T21:36:16.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53642025-05-30T21:36:16.635ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53652025-05-30T21:36:16.639ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
53662025-05-30T21:36:16.639ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53672025-05-30T21:36:16.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53682025-05-30T21:36:16.645ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 2990b628-13b9-47b6-8f08-4b84713240a8, kind: Service { id: 2a161087-f464-41f8-b020-0f9e83fd5351 }, name: Name("external-dns-2a161087-f464-41f8-b020-0f9e83fd5351"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 127])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 535411d1-cf1c-44db-becb-5ce60a31b05b, kind: Service { id: 5c275d18-0bbd-414f-a1ca-4f5fb261fa3a }, name: Name("external-dns-5c275d18-0bbd-414f-a1ca-4f5fb261fa3a"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 225, 1])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 957a8a3c-41fc-4640-b220-a063b1626528, kind: Service { id: 8afbed31-e520-4cc6-aa5c-72f8b96b171a }, name: Name("nexus-8afbed31-e520-4cc6-aa5c-72f8b96b171a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 83])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b2cee132-ec62-413c-b43c-0e9042d338c2, kind: Service { id: bb2ce2e9-85b4-4cff-aedf-120e3ffadec2 }, name: Name("nexus-bb2ce2e9-85b4-4cff-aedf-120e3ffadec2"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 208, 170])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bea3c463-5b2a-46b8-ad8c-4e1a767d4ece, kind: Service { id: ace9a692-9e36-4a57-88cc-937ba7a7ab5e }, name: Name("nexus-ace9a692-9e36-4a57-88cc-937ba7a7ab5e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
53692025-05-30T21:36:16.645ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53702025-05-30T21:36:16.646ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53712025-05-30T21:36:16.649ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
53722025-05-30T21:36:16.649ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53732025-05-30T21:36:16.650ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "eab80f63-4bd6-4baf-bdcf-efda200dc551", "date": "Fri, 30 May 2025 21:36:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
53742025-05-30T21:36:16.650ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
53752025-05-30T21:36:16.652ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
53762025-05-30T21:36:16.652ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53772025-05-30T21:36:16.652ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53782025-05-30T21:36:16.652ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "48c30d04-c256-48ab-97a6-c50cc23e68f6", "date": "Fri, 30 May 2025 21:36:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
53792025-05-30T21:36:16.652ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
53802025-05-30T21:36:16.654ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
53812025-05-30T21:36:16.654ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 2990b628-13b9-47b6-8f08-4b84713240a8, kind: Service { id: 2a161087-f464-41f8-b020-0f9e83fd5351 }, name: Name("external-dns-2a161087-f464-41f8-b020-0f9e83fd5351"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 127])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 535411d1-cf1c-44db-becb-5ce60a31b05b, kind: Service { id: 5c275d18-0bbd-414f-a1ca-4f5fb261fa3a }, name: Name("external-dns-5c275d18-0bbd-414f-a1ca-4f5fb261fa3a"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 225, 1])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 957a8a3c-41fc-4640-b220-a063b1626528, kind: Service { id: 8afbed31-e520-4cc6-aa5c-72f8b96b171a }, name: Name("nexus-8afbed31-e520-4cc6-aa5c-72f8b96b171a"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 83])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b2cee132-ec62-413c-b43c-0e9042d338c2, kind: Service { id: bb2ce2e9-85b4-4cff-aedf-120e3ffadec2 }, name: Name("nexus-bb2ce2e9-85b4-4cff-aedf-120e3ffadec2"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 208, 170])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bea3c463-5b2a-46b8-ad8c-4e1a767d4ece, kind: Service { id: ace9a692-9e36-4a57-88cc-937ba7a7ab5e }, name: Name("nexus-ace9a692-9e36-4a57-88cc-937ba7a7ab5e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
53822025-05-30T21:36:16.654ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
53832025-05-30T21:36:16.654ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "7926698f-2b38-4391-87c2-47d2935806ca", "date": "Fri, 30 May 2025 21:36:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
53842025-05-30T21:36:16.654ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
53852025-05-30T21:36:16.656ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
53862025-05-30T21:36:16.656ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 715.079833ms iteration = 13 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
53872025-05-30T21:36:16.689ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
53882025-05-30T21:36:16.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
53892025-05-30T21:36:16.691ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
53902025-05-30T21:36:16.691ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
53912025-05-30T21:36:16.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
53922025-05-30T21:36:16.705ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
53932025-05-30T21:36:16.705ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
53942025-05-30T21:36:16.706ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
53952025-05-30T21:36:16.706ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
53962025-05-30T21:36:16.706ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
53972025-05-30T21:36:16.706ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
53982025-05-30T21:36:16.706ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
53992025-05-30T21:36:16.706ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
54002025-05-30T21:36:16.706ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
54012025-05-30T21:36:16.706ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
54022025-05-30T21:36:16.706ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
54032025-05-30T21:36:16.706ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54042025-05-30T21:36:16.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54052025-05-30T21:36:16.712ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
54062025-05-30T21:36:16.712ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
54072025-05-30T21:36:16.786ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 5d926227-6a52-4938-b74d-21ba39bce2f9, time_created: 2025-05-30T21:26:16.127586Z, time_modified: 2025-05-30T21:26:16.127586Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(56094621696)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
54082025-05-30T21:36:16.786ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
54092025-05-30T21:36:16.820ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54102025-05-30T21:36:16.820ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54112025-05-30T21:36:16.823ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54122025-05-30T21:36:16.823ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54132025-05-30T21:36:16.824ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54142025-05-30T21:36:16.826ZDEBGnexus (BackgroundTasks): client response SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7168cf4-4c69-48aa-9986-b2e152250183", "content-length": "51811", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
54152025-05-30T21:36:16.827ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
54162025-05-30T21:36:16.827ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
54172025-05-30T21:36:16.828ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "34b2c279-e2d3-4773-b28a-e7bdcc00a136", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
54182025-05-30T21:36:16.828ZDEBGnexus (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
54192025-05-30T21:36:16.828ZDEBGnexus (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
54202025-05-30T21:36:16.828ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
54212025-05-30T21:36:16.830ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:495
54222025-05-30T21:36:16.830ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 888.605156ms
54232025-05-30T21:36:16.830ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 888.613826ms iteration = 3 status = {}
54242025-05-30T21:36:16.830ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54252025-05-30T21:36:16.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54262025-05-30T21:36:16.832ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54272025-05-30T21:36:16.834ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
54282025-05-30T21:36:16.834ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54292025-05-30T21:36:16.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54302025-05-30T21:36:16.929ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "dff9d68c-1cb8-4475-b8e5-24f0b449c686", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
54312025-05-30T21:36:16.929ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
54322025-05-30T21:36:16.929ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
54332025-05-30T21:36:16.929ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d6f6938-5780-4b31-88f2-cfab302f2c53", "content-length": "682", "date": "Fri, 30 May 2025 21:36:16 GMT"} })
54342025-05-30T21:36:16.929ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54352025-05-30T21:36:16.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54362025-05-30T21:36:16.933ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54372025-05-30T21:36:16.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54382025-05-30T21:36:16.936ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54392025-05-30T21:36:16.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54402025-05-30T21:36:16.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54412025-05-30T21:36:16.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54422025-05-30T21:36:16.946ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 1.005099705s iteration = 21 status = {}
54432025-05-30T21:36:17.158ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 0413f1aa-e684-44c2-a1e9-1a20a4631bb4 file = nexus/db-queries/src/db/datastore/inventory.rs:1192
54442025-05-30T21:36:17.158ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 0413f1aa-e684-44c2-a1e9-1a20a4631bb4 time_started = 2025-05-30 21:36:16.258821 UTC
54452025-05-30T21:36:17.158ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 5 reason = Dependency
54462025-05-30T21:36:17.158ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.217314082s iteration = 2 status = {"collection_id":"0413f1aa-e684-44c2-a1e9-1a20a4631bb4","time_done":"2025-05-30 21:36:16.828582 UTC","time_started":"2025-05-30 21:36:16.258821 UTC"}
54472025-05-30T21:36:17.158ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 23 reason = Dependency
54482025-05-30T21:36:17.158ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54492025-05-30T21:36:17.158ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
54502025-05-30T21:36:17.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54512025-05-30T21:36:17.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
54522025-05-30T21:36:17.160ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54532025-05-30T21:36:17.160ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54542025-05-30T21:36:17.162ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
54552025-05-30T21:36:17.163ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
54562025-05-30T21:36:17.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
54572025-05-30T21:36:17.164ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
54582025-05-30T21:36:17.164ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54592025-05-30T21:36:17.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54602025-05-30T21:36:17.164ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
54612025-05-30T21:36:17.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
54622025-05-30T21:36:17.166ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54632025-05-30T21:36:17.168ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
54642025-05-30T21:36:17.168ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
54652025-05-30T21:36:17.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
54662025-05-30T21:36:17.210ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54672025-05-30T21:36:17.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54682025-05-30T21:36:17.236ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 77.206077ms iteration = 23 status = {"physical_disks_added":0}
54692025-05-30T21:36:17.290ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54702025-05-30T21:36:17.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54712025-05-30T21:36:17.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54722025-05-30T21:36:17.293ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
54732025-05-30T21:36:17.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54742025-05-30T21:36:17.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54752025-05-30T21:36:17.295ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54762025-05-30T21:36:17.296ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
54772025-05-30T21:36:17.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54782025-05-30T21:36:17.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54792025-05-30T21:36:17.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54802025-05-30T21:36:17.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54812025-05-30T21:36:17.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54822025-05-30T21:36:17.302ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
54832025-05-30T21:36:17.302ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54842025-05-30T21:36:17.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54852025-05-30T21:36:17.303ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54862025-05-30T21:36:17.304ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
54872025-05-30T21:36:17.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54882025-05-30T21:36:17.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54892025-05-30T21:36:17.307ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
54902025-05-30T21:36:17.307ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 148.415655ms iteration = 5 status = {"blueprint_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","inventory_collection_id":"0413f1aa-e684-44c2-a1e9-1a20a4631bb4","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
54912025-05-30T21:36:18.777ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 327 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = b420e286-e506-4b22-bc59-4094c77cb3a9 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
54922025-05-30T21:36:25.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 62 reason = Timeout
54932025-05-30T21:36:25.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 62 reason = Timeout
54942025-05-30T21:36:25.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54952025-05-30T21:36:25.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
54962025-05-30T21:36:25.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54972025-05-30T21:36:25.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
54982025-05-30T21:36:25.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54992025-05-30T21:36:25.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
55002025-05-30T21:36:25.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55012025-05-30T21:36:25.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55022025-05-30T21:36:25.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55032025-05-30T21:36:25.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55042025-05-30T21:36:25.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55052025-05-30T21:36:25.952ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
55062025-05-30T21:36:25.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55072025-05-30T21:36:25.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55082025-05-30T21:36:25.962ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
55092025-05-30T21:36:25.962ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.815249ms iteration = 62 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
55102025-05-30T21:36:25.973ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55112025-05-30T21:36:25.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55122025-05-30T21:36:26.005ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.44439ms iteration = 62 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
55132025-05-30T21:36:28.777ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 363 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 2897d1bb-3c34-4a12-a98d-c25eb5768a7a response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
55142025-05-30T21:36:35.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 63 reason = Timeout
55152025-05-30T21:36:35.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 63 reason = Timeout
55162025-05-30T21:36:35.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55172025-05-30T21:36:35.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55182025-05-30T21:36:35.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55192025-05-30T21:36:35.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55202025-05-30T21:36:35.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55212025-05-30T21:36:35.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55222025-05-30T21:36:35.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55232025-05-30T21:36:35.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
55242025-05-30T21:36:35.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55252025-05-30T21:36:35.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55262025-05-30T21:36:35.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55272025-05-30T21:36:35.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55282025-05-30T21:36:35.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55292025-05-30T21:36:35.953ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
55302025-05-30T21:36:35.953ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55312025-05-30T21:36:35.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55322025-05-30T21:36:35.964ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
55332025-05-30T21:36:35.964ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.688141ms iteration = 63 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
55342025-05-30T21:36:35.975ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.228393ms iteration = 63 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
55352025-05-30T21:36:38.777ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 314 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 837c2cae-5ec0-41fb-99b5-652c565b10f7 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
55362025-05-30T21:36:43.767ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:44893
55372025-05-30T21:36:43.767ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44893 req_id = 6f2633f5-7947-4da2-b98f-e5c8c4684d82 roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
55382025-05-30T21:36:43.767ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44893 req_id = 6f2633f5-7947-4da2-b98f-e5c8c4684d82 resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
55392025-05-30T21:36:43.769ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2280 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44893 req_id = 6f2633f5-7947-4da2-b98f-e5c8c4684d82 response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
55402025-05-30T21:36:43.769ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44893 req_id = 906969cd-b498-435a-a144-1bd2f0b1148a roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
55412025-05-30T21:36:43.770ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44893 req_id = 906969cd-b498-435a-a144-1bd2f0b1148a resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
55422025-05-30T21:36:43.772ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2425 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44893 req_id = 906969cd-b498-435a-a144-1bd2f0b1148a response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
55432025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 64 reason = Timeout
55442025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 22 reason = Timeout
55452025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55462025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 22 reason = Timeout
55472025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
55482025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
55492025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 22 reason = Timeout
55502025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
55512025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 22 reason = Timeout
55522025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 22 reason = Timeout
55532025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
55542025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
55552025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 22 reason = Timeout
55562025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
55572025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 22 reason = Timeout
55582025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
55592025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
55602025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 22 reason = Timeout
55612025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
55622025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 22 reason = Timeout
55632025-05-30T21:36:45.942ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
55642025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
55652025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 22 reason = Timeout
55662025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
55672025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 22 reason = Timeout
55682025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55692025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 22 reason = Timeout
55702025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
55712025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 22 reason = Timeout
55722025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
55732025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 22 reason = Timeout
55742025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
55752025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 24 reason = Timeout
55762025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
55772025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 22 reason = Timeout
55782025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
55792025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 64 reason = Timeout
55802025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55812025-05-30T21:36:45.942ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 22 reason = Timeout
55822025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
55832025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
55842025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
55852025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
55862025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55872025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
55882025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
55892025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
55902025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
55912025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
55922025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
55932025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55942025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
55952025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55962025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
55972025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
55982025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
55992025-05-30T21:36:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
56002025-05-30T21:36:45.945ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1740, tv_nsec: 881681517 } })
56012025-05-30T21:36:45.945ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56022025-05-30T21:36:45.945ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
56032025-05-30T21:36:45.946ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 3.726383ms iteration = 22 status = {"errors":[],"finish_invoked_ok":[]}
56042025-05-30T21:36:45.946ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.141457ms iteration = 22 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
56052025-05-30T21:36:45.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
56062025-05-30T21:36:45.948ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56072025-05-30T21:36:45.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56082025-05-30T21:36:45.950ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56092025-05-30T21:36:45.950ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56102025-05-30T21:36:45.951ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56112025-05-30T21:36:45.951ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56122025-05-30T21:36:45.951ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
56132025-05-30T21:36:45.952ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56142025-05-30T21:36:45.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
56152025-05-30T21:36:45.953ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
56162025-05-30T21:36:45.953ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
56172025-05-30T21:36:45.953ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
56182025-05-30T21:36:45.954ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56192025-05-30T21:36:45.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56202025-05-30T21:36:45.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
56212025-05-30T21:36:45.954ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
56222025-05-30T21:36:45.954ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56232025-05-30T21:36:45.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56242025-05-30T21:36:45.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
56252025-05-30T21:36:45.955ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56262025-05-30T21:36:45.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
56272025-05-30T21:36:45.955ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56282025-05-30T21:36:45.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56292025-05-30T21:36:45.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56302025-05-30T21:36:45.955ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56312025-05-30T21:36:45.957ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1740, tv_nsec: 893219770 } })
56322025-05-30T21:36:45.957ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
56332025-05-30T21:36:45.958ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
56342025-05-30T21:36:45.958ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56352025-05-30T21:36:45.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56362025-05-30T21:36:45.963ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 20.98309ms iteration = 22 status = {"errors":[],"garbage_collect_requested":[]}
56372025-05-30T21:36:45.963ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
56382025-05-30T21:36:45.963ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
56392025-05-30T21:36:45.963ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
56402025-05-30T21:36:45.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
56412025-05-30T21:36:45.964ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56422025-05-30T21:36:45.964ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56432025-05-30T21:36:45.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56442025-05-30T21:36:45.966ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
56452025-05-30T21:36:45.966ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56462025-05-30T21:36:45.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56472025-05-30T21:36:45.967ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56482025-05-30T21:36:45.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56492025-05-30T21:36:45.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56502025-05-30T21:36:45.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56512025-05-30T21:36:45.969ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1740, tv_nsec: 906042957 } })
56522025-05-30T21:36:45.970ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
56532025-05-30T21:36:45.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56542025-05-30T21:36:45.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
56552025-05-30T21:36:45.975ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56562025-05-30T21:36:45.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56572025-05-30T21:36:45.988ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
56582025-05-30T21:36:45.988ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 45.21034ms iteration = 64 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
56592025-05-30T21:36:45.989ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56602025-05-30T21:36:45.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56612025-05-30T21:36:45.990ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
56622025-05-30T21:36:45.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
56632025-05-30T21:36:45.994ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56642025-05-30T21:36:45.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56652025-05-30T21:36:45.997ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56662025-05-30T21:36:45.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56672025-05-30T21:36:45.998ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56682025-05-30T21:36:46.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
56692025-05-30T21:36:46.003ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56702025-05-30T21:36:46.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56712025-05-30T21:36:46.005ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 63.169066ms iteration = 22 status = 0
56722025-05-30T21:36:46.009ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56732025-05-30T21:36:46.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56742025-05-30T21:36:46.010ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 67.359944ms iteration = 22 status = {"errors":[],"requests_created_ok":[]}
56752025-05-30T21:36:46.010ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
56762025-05-30T21:36:46.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
56772025-05-30T21:36:46.011ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56782025-05-30T21:36:46.012ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
56792025-05-30T21:36:46.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
56802025-05-30T21:36:46.013ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
56812025-05-30T21:36:46.013ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56822025-05-30T21:36:46.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56832025-05-30T21:36:46.014ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
56842025-05-30T21:36:46.014ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
56852025-05-30T21:36:46.014ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
56862025-05-30T21:36:46.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
56872025-05-30T21:36:46.017ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 74.752298ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
56882025-05-30T21:36:46.017ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
56892025-05-30T21:36:46.017ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 74.89167ms iteration = 22 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
56902025-05-30T21:36:46.025ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56912025-05-30T21:36:46.026ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
56922025-05-30T21:36:46.026ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
56932025-05-30T21:36:46.026ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a988fb77-b117-4735-b235-339941071aa2", "content-length": "26", "date": "Fri, 30 May 2025 21:36:45 GMT"} })
56942025-05-30T21:36:46.026ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
56952025-05-30T21:36:46.027ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
56962025-05-30T21:36:46.027ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56972025-05-30T21:36:46.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56982025-05-30T21:36:46.031ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
56992025-05-30T21:36:46.032ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 89.557069ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
57002025-05-30T21:36:46.037ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 95.452686ms iteration = 64 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
57012025-05-30T21:36:46.061ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 118.535921ms iteration = 22 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
57022025-05-30T21:36:46.061ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
57032025-05-30T21:36:46.061ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
57042025-05-30T21:36:46.061ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 118.913396ms iteration = 22 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
57052025-05-30T21:36:46.068ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
57062025-05-30T21:36:46.068ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57072025-05-30T21:36:46.069ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd64ee45-f3d5-4bbb-96b1-e55d093da39d", "content-length": "26", "date": "Fri, 30 May 2025 21:36:46 GMT"} })
57082025-05-30T21:36:46.069ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57092025-05-30T21:36:46.070ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
57102025-05-30T21:36:46.073ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
57112025-05-30T21:36:46.073ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 130.255746ms iteration = 22 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
57122025-05-30T21:36:46.097ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57132025-05-30T21:36:46.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57142025-05-30T21:36:46.114ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
57152025-05-30T21:36:46.114ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57162025-05-30T21:36:46.114ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e70e8d1-5255-4ffb-b345-087942fb1dba", "content-length": "26", "date": "Fri, 30 May 2025 21:36:46 GMT"} })
57172025-05-30T21:36:46.114ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57182025-05-30T21:36:46.116ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
57192025-05-30T21:36:46.170ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 227.738376ms iteration = 24 status = {"physical_disks_added":0}
57202025-05-30T21:36:46.177ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
57212025-05-30T21:36:46.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
57222025-05-30T21:36:46.184ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
57232025-05-30T21:36:46.184ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
57242025-05-30T21:36:46.184ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 241.920289ms iteration = 22 status = {}
57252025-05-30T21:36:46.184ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "27541f2e-bee0-4f93-a396-9ce123e51377", "content-length": "372", "date": "Fri, 30 May 2025 21:36:45 GMT"} })
57262025-05-30T21:36:46.184ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57272025-05-30T21:36:46.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57282025-05-30T21:36:46.186ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 243.563618ms iteration = 22 status = null
57292025-05-30T21:36:46.225ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57302025-05-30T21:36:46.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57312025-05-30T21:36:46.227ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57322025-05-30T21:36:46.229ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
57332025-05-30T21:36:46.229ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57342025-05-30T21:36:46.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57352025-05-30T21:36:46.230ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57362025-05-30T21:36:46.231ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
57372025-05-30T21:36:46.231ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
57382025-05-30T21:36:46.231ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "5f8f424d-294c-486c-bb0c-c58057f6840c", "date": "Fri, 30 May 2025 21:36:45 GMT"} })
57392025-05-30T21:36:46.231ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57402025-05-30T21:36:46.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57412025-05-30T21:36:46.232ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
57422025-05-30T21:36:46.232ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57432025-05-30T21:36:46.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57442025-05-30T21:36:46.234ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57452025-05-30T21:36:46.234ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57462025-05-30T21:36:46.236ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57472025-05-30T21:36:46.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57482025-05-30T21:36:46.239ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
57492025-05-30T21:36:46.239ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
57502025-05-30T21:36:46.239ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "0ae8bfda-4ad6-4ced-9694-a753ef14d5d4", "date": "Fri, 30 May 2025 21:36:45 GMT"} })
57512025-05-30T21:36:46.241ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 298.847202ms iteration = 22 status = {}
57522025-05-30T21:36:46.273ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57532025-05-30T21:36:46.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57542025-05-30T21:36:46.276ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
57552025-05-30T21:36:46.276ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57562025-05-30T21:36:46.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57572025-05-30T21:36:46.285ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
57582025-05-30T21:36:46.286ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e592794e-76de-4d54-bd4f-00016f68992d", "content-length": "691", "date": "Fri, 30 May 2025 21:36:45 GMT"} })
57592025-05-30T21:36:46.286ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
57602025-05-30T21:36:46.286ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
57612025-05-30T21:36:46.286ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
57622025-05-30T21:36:46.287ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 345.022163ms iteration = 22 status = {}
57632025-05-30T21:36:46.288ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
57642025-05-30T21:36:46.288ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
57652025-05-30T21:36:46.288ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
57662025-05-30T21:36:46.288ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
57672025-05-30T21:36:46.288ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
57682025-05-30T21:36:46.288ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
57692025-05-30T21:36:46.288ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
57702025-05-30T21:36:46.288ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
57712025-05-30T21:36:46.288ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
57722025-05-30T21:36:46.289ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
57732025-05-30T21:36:46.289ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
57742025-05-30T21:36:46.289ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57752025-05-30T21:36:46.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57762025-05-30T21:36:46.291ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
57772025-05-30T21:36:46.291ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
57782025-05-30T21:36:46.400ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "2e9b443d-c3e2-4e6b-a5cc-0d9c12fc1f6b", "date": "Fri, 30 May 2025 21:36:45 GMT"} })
57792025-05-30T21:36:46.400ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
57802025-05-30T21:36:46.400ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
57812025-05-30T21:36:46.400ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "4733a353-a0c4-4c0d-b07d-19cb65d6ab34", "content-length": "682", "date": "Fri, 30 May 2025 21:36:46 GMT"} })
57822025-05-30T21:36:46.400ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57832025-05-30T21:36:46.401ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57842025-05-30T21:36:46.404ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57852025-05-30T21:36:46.404ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57862025-05-30T21:36:46.408ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57872025-05-30T21:36:46.408ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57882025-05-30T21:36:46.410ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57892025-05-30T21:36:46.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57902025-05-30T21:36:46.417ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 474.53265ms iteration = 22 status = {}
57912025-05-30T21:36:48.777ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 339 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = aa4118a4-04e5-4d50-a5cd-47d29bb01637 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
57922025-05-30T21:36:55.941ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 65 reason = Timeout
57932025-05-30T21:36:55.941ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
57942025-05-30T21:36:55.941ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 65 reason = Timeout
57952025-05-30T21:36:55.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57962025-05-30T21:36:55.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
57972025-05-30T21:36:55.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57982025-05-30T21:36:55.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
57992025-05-30T21:36:55.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58002025-05-30T21:36:55.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58012025-05-30T21:36:55.945ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.627512ms iteration = 65 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
58022025-05-30T21:36:55.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
58032025-05-30T21:36:55.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58042025-05-30T21:36:55.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58052025-05-30T21:36:55.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58062025-05-30T21:36:55.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58072025-05-30T21:36:55.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58082025-05-30T21:36:55.953ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
58092025-05-30T21:36:55.953ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58102025-05-30T21:36:55.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58112025-05-30T21:36:55.961ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
58122025-05-30T21:36:55.961ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.872379ms iteration = 65 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
58132025-05-30T21:36:58.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 344 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 655826f6-1c63-4d2e-be6a-bb552e9db988 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
58142025-05-30T21:37:05.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 66 reason = Timeout
58152025-05-30T21:37:05.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58162025-05-30T21:37:05.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 66 reason = Timeout
58172025-05-30T21:37:05.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58182025-05-30T21:37:05.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58192025-05-30T21:37:05.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58202025-05-30T21:37:05.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58212025-05-30T21:37:05.945ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58222025-05-30T21:37:05.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58232025-05-30T21:37:05.947ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.948337ms iteration = 66 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
58242025-05-30T21:37:05.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
58252025-05-30T21:37:05.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58262025-05-30T21:37:05.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58272025-05-30T21:37:05.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58282025-05-30T21:37:05.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58292025-05-30T21:37:05.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58302025-05-30T21:37:05.954ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
58312025-05-30T21:37:05.954ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58322025-05-30T21:37:05.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58332025-05-30T21:37:05.965ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
58342025-05-30T21:37:05.965ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.823892ms iteration = 66 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
58352025-05-30T21:37:08.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 344 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 61b0a18f-3bab-421b-ad03-437dc97e838b response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
58362025-05-30T21:37:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 12 reason = Timeout
58372025-05-30T21:37:15.941ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
58382025-05-30T21:37:15.941ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
58392025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
58402025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 67 reason = Timeout
58412025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58422025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 23 reason = Timeout
58432025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 23 reason = Timeout
58442025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 13 reason = Timeout
58452025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
58462025-05-30T21:37:15.942ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e
58472025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
58482025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.72\u{b5}s iteration = 13 status = {"enabled":false,"target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e"}
58492025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 12 reason = Timeout
58502025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 23 reason = Timeout
58512025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 23 reason = Timeout
58522025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
58532025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
58542025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 12 reason = Timeout
58552025-05-30T21:37:15.942ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
58562025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 23 reason = Timeout
58572025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
58582025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
58592025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 23 reason = Timeout
58602025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
58612025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 12 reason = Timeout
58622025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 12 reason = Timeout
58632025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
58642025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
58652025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
58662025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 12 reason = Timeout
58672025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
58682025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 12 reason = Timeout
58692025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
58702025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 23 reason = Timeout
58712025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58722025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 23 reason = Timeout
58732025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
58742025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58752025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 14 reason = Timeout
58762025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 23 reason = Timeout
58772025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58782025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 67 reason = Timeout
58792025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 23 reason = Timeout
58802025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58812025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 12 reason = Timeout
58822025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
58832025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 23 reason = Timeout
58842025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
58852025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 12 reason = Timeout
58862025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
58872025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 12 reason = Timeout
58882025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
58892025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 12 reason = Timeout
58902025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 14 reason = Timeout
58912025-05-30T21:37:15.942ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-30 21:27:15.942412814 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
58922025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 23 reason = Timeout
58932025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
58942025-05-30T21:37:15.942ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
58952025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58962025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 23 reason = Timeout
58972025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 12 reason = Timeout
58982025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
58992025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
59002025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 25 reason = Timeout
59012025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
59022025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 23 reason = Timeout
59032025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
59042025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 23 reason = Timeout
59052025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
59062025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 23 reason = Timeout
59072025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
59082025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
59092025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
59102025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
59112025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
59122025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59132025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
59142025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
59152025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
59162025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
59172025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
59182025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
59192025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
59202025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59212025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
59222025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
59232025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
59242025-05-30T21:37:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
59252025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
59262025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
59272025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
59282025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59292025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
59302025-05-30T21:37:15.943ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2a161087-f464-41f8-b020-0f9e83fd5351.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1770, tv_nsec: 878977781 } })
59312025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
59322025-05-30T21:37:15.943ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0413f6b4-86a1-42a1-919a-e8f42ff9f638.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("acb1452d-0a02-4b36-a462-c8ce2f96a488.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:2::1))) }], valid_until: Instant { tv_sec: 1770, tv_nsec: 878999291 } })
59332025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
59342025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
59352025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
59362025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
59372025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
59382025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
59392025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.382476ms iteration = 12 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
59402025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } dns_group = internal
59412025-05-30T21:37:15.943ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.270485ms iteration = 12 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353"]}
59422025-05-30T21:37:15.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59432025-05-30T21:37:15.945ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.826604ms iteration = 23 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
59442025-05-30T21:37:15.946ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1770, tv_nsec: 882238429 } })
59452025-05-30T21:37:15.946ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
59462025-05-30T21:37:15.946ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
59472025-05-30T21:37:15.946ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
59482025-05-30T21:37:15.946ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
59492025-05-30T21:37:15.946ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59502025-05-30T21:37:15.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
59512025-05-30T21:37:15.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
59522025-05-30T21:37:15.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59532025-05-30T21:37:15.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59542025-05-30T21:37:15.949ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
59552025-05-30T21:37:15.949ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
59562025-05-30T21:37:15.949ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59572025-05-30T21:37:15.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
59582025-05-30T21:37:15.949ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
59592025-05-30T21:37:15.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59602025-05-30T21:37:15.949ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 7.125891ms iteration = 23 status = {"errors":[],"finish_invoked_ok":[]}
59612025-05-30T21:37:15.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
59622025-05-30T21:37:15.951ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:202 num_receivers = 0
59632025-05-30T21:37:15.951ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 9.697051ms iteration = 12 status = {"by_rx":{},"error":null}
59642025-05-30T21:37:15.952ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59652025-05-30T21:37:15.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59662025-05-30T21:37:15.952ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59672025-05-30T21:37:15.952ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 10.716633ms iteration = 12 status = {"errors":[],"found_port_ok":[]}
59682025-05-30T21:37:15.953ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
59692025-05-30T21:37:15.953ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
59702025-05-30T21:37:15.953ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 11.120208ms iteration = 12 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
59712025-05-30T21:37:15.954ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
59722025-05-30T21:37:15.954ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
59732025-05-30T21:37:15.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
59742025-05-30T21:37:15.956ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59752025-05-30T21:37:15.956ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59762025-05-30T21:37:15.957ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 14.79668ms iteration = 23 status = {"errors":[],"garbage_collect_requested":[]}
59772025-05-30T21:37:15.958ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1770, tv_nsec: 894530830 } })
59782025-05-30T21:37:15.958ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
59792025-05-30T21:37:15.959ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
59802025-05-30T21:37:15.959ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59812025-05-30T21:37:15.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
59822025-05-30T21:37:15.959ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
59832025-05-30T21:37:15.959ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59842025-05-30T21:37:15.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59852025-05-30T21:37:15.961ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-30 21:27:15.942412814 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
59862025-05-30T21:37:15.961ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 18.723885ms iteration = 12 status = {"expiration":"2025-05-30T21:27:15.942412814Z","pruned":[]}
59872025-05-30T21:37:15.961ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59882025-05-30T21:37:15.963ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59892025-05-30T21:37:15.963ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
59902025-05-30T21:37:15.963ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
59912025-05-30T21:37:15.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
59922025-05-30T21:37:15.965ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59932025-05-30T21:37:15.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59942025-05-30T21:37:15.965ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
59952025-05-30T21:37:15.965ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
59962025-05-30T21:37:15.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
59972025-05-30T21:37:15.966ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
59982025-05-30T21:37:15.966ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
59992025-05-30T21:37:15.966ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
60002025-05-30T21:37:15.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
60012025-05-30T21:37:15.967ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1770, tv_nsec: 903390521 } })
60022025-05-30T21:37:15.967ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
60032025-05-30T21:37:15.968ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60042025-05-30T21:37:15.968ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60052025-05-30T21:37:15.968ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60062025-05-30T21:37:15.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60072025-05-30T21:37:15.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60082025-05-30T21:37:15.969ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
60092025-05-30T21:37:15.969ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
60102025-05-30T21:37:15.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
60112025-05-30T21:37:15.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
60122025-05-30T21:37:15.971ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
60132025-05-30T21:37:15.971ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60142025-05-30T21:37:15.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60152025-05-30T21:37:15.972ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
60162025-05-30T21:37:15.973ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
60172025-05-30T21:37:15.973ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60182025-05-30T21:37:15.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60192025-05-30T21:37:15.974ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
60202025-05-30T21:37:15.974ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
60212025-05-30T21:37:15.974ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
60222025-05-30T21:37:15.974ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
60232025-05-30T21:37:15.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
60242025-05-30T21:37:15.975ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60252025-05-30T21:37:15.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60262025-05-30T21:37:15.975ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 33.553055ms iteration = 12 status = {"dispatched":[],"errors":[],"glob_version":"144.0.0","globs_reprocessed":{},"no_receivers":[]}
60272025-05-30T21:37:15.976ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60282025-05-30T21:37:15.978ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60292025-05-30T21:37:15.982ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
60302025-05-30T21:37:15.982ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
60312025-05-30T21:37:15.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
60322025-05-30T21:37:15.983ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
60332025-05-30T21:37:15.983ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
60342025-05-30T21:37:15.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
60352025-05-30T21:37:15.987ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
60362025-05-30T21:37:15.987ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 45.498593ms iteration = 67 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
60372025-05-30T21:37:15.990ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60382025-05-30T21:37:15.993ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 50.686193ms iteration = 23 status = {"errors":[],"requests_created_ok":[]}
60392025-05-30T21:37:15.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
60402025-05-30T21:37:15.993ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
60412025-05-30T21:37:15.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60422025-05-30T21:37:15.993ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
60432025-05-30T21:37:15.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
60442025-05-30T21:37:16.000ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
60452025-05-30T21:37:16.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
60462025-05-30T21:37:16.001ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60472025-05-30T21:37:16.004ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
60482025-05-30T21:37:16.004ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
60492025-05-30T21:37:16.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
60502025-05-30T21:37:16.004ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
60512025-05-30T21:37:16.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
60522025-05-30T21:37:16.006ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 63.999555ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
60532025-05-30T21:37:16.006ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60542025-05-30T21:37:16.007ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60552025-05-30T21:37:16.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60562025-05-30T21:37:16.008ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
60572025-05-30T21:37:16.008ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
60582025-05-30T21:37:16.008ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 66.325812ms iteration = 12 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
60592025-05-30T21:37:16.009ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
60602025-05-30T21:37:16.009ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 67.479595ms iteration = 67 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
60612025-05-30T21:37:16.010ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
60622025-05-30T21:37:16.010ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
60632025-05-30T21:37:16.010ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
60642025-05-30T21:37:16.010ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
60652025-05-30T21:37:16.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
60662025-05-30T21:37:16.020ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
60672025-05-30T21:37:16.020ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
60682025-05-30T21:37:16.020ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 77.945275ms iteration = 23 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
60692025-05-30T21:37:16.074ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
60702025-05-30T21:37:16.076ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
60712025-05-30T21:37:16.086ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
60722025-05-30T21:37:16.086ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
60732025-05-30T21:37:16.086ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "16bd4745-c484-483f-99ce-69c6d25f651b", "content-length": "26", "date": "Fri, 30 May 2025 21:37:16 GMT"} })
60742025-05-30T21:37:16.086ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
60752025-05-30T21:37:16.087ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
60762025-05-30T21:37:16.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
60772025-05-30T21:37:16.088ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
60782025-05-30T21:37:16.095ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
60792025-05-30T21:37:16.095ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
60802025-05-30T21:37:16.095ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "febb7963-b55d-4537-be62-20aead24473a", "content-length": "26", "date": "Fri, 30 May 2025 21:37:16 GMT"} })
60812025-05-30T21:37:16.095ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
60822025-05-30T21:37:16.097ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
60832025-05-30T21:37:16.138ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
60842025-05-30T21:37:16.138ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
60852025-05-30T21:37:16.138ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4e3f970-433a-4a0b-b9b3-f87882f258d5", "content-length": "372", "date": "Fri, 30 May 2025 21:37:15 GMT"} })
60862025-05-30T21:37:16.138ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60872025-05-30T21:37:16.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60882025-05-30T21:37:16.141ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
60892025-05-30T21:37:16.141ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
60902025-05-30T21:37:16.141ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "27355637-dd58-4b87-95bc-e89bcfd78273", "content-length": "26", "date": "Fri, 30 May 2025 21:37:16 GMT"} })
60912025-05-30T21:37:16.141ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
60922025-05-30T21:37:16.143ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60932025-05-30T21:37:16.143ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
60942025-05-30T21:37:16.146ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
60952025-05-30T21:37:16.146ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60962025-05-30T21:37:16.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60972025-05-30T21:37:16.163ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60982025-05-30T21:37:16.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60992025-05-30T21:37:16.204ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
61002025-05-30T21:37:16.206ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
61012025-05-30T21:37:16.206ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
61022025-05-30T21:37:16.206ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
61032025-05-30T21:37:16.207ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
61042025-05-30T21:37:16.208ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
61052025-05-30T21:37:16.209ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
61062025-05-30T21:37:16.209ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
61072025-05-30T21:37:16.209ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
61082025-05-30T21:37:16.210ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
61092025-05-30T21:37:16.211ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 268.543373ms iteration = 23 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
61102025-05-30T21:37:16.212ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61112025-05-30T21:37:16.213ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61122025-05-30T21:37:16.216ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
61132025-05-30T21:37:16.216ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
61142025-05-30T21:37:16.216ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
61152025-05-30T21:37:16.216ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
61162025-05-30T21:37:16.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
61172025-05-30T21:37:16.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
61182025-05-30T21:37:16.218ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
61192025-05-30T21:37:16.218ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 275.514734ms iteration = 23 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
61202025-05-30T21:37:16.218ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
61212025-05-30T21:37:16.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
61222025-05-30T21:37:16.222ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 280.103016ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
61232025-05-30T21:37:16.223ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61242025-05-30T21:37:16.224ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61252025-05-30T21:37:16.226ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
61262025-05-30T21:37:16.226ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
61272025-05-30T21:37:16.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
61282025-05-30T21:37:16.228ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 286.399369ms iteration = 23 status = {}
61292025-05-30T21:37:16.229ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 286.675242ms iteration = 23 status = 0
61302025-05-30T21:37:16.230ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
61312025-05-30T21:37:16.230ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
61322025-05-30T21:37:16.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
61332025-05-30T21:37:16.240ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
61342025-05-30T21:37:16.240ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
61352025-05-30T21:37:16.242ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 300.280088ms iteration = 23 status = null
61362025-05-30T21:37:16.243ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
61372025-05-30T21:37:16.243ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "e4ef99c6-29e2-447d-b576-c25b78d8d219", "date": "Fri, 30 May 2025 21:37:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
61382025-05-30T21:37:16.243ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
61392025-05-30T21:37:16.245ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "0f40f21e-0bc8-4c84-9ec9-aba68f18f208", "date": "Fri, 30 May 2025 21:37:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
61402025-05-30T21:37:16.245ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
61412025-05-30T21:37:16.247ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
61422025-05-30T21:37:16.247ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 305.581969ms iteration = 14 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
61432025-05-30T21:37:16.256ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
61442025-05-30T21:37:16.256ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
61452025-05-30T21:37:16.257ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "c6451dbb-f05e-4f5c-a501-babf721d1970", "date": "Fri, 30 May 2025 21:37:15 GMT"} })
61462025-05-30T21:37:16.257ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61472025-05-30T21:37:16.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61482025-05-30T21:37:16.259ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61492025-05-30T21:37:16.259ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61502025-05-30T21:37:16.260ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-30T21:26:13.986396Z, time_modified: 2025-05-30T21:26:13.986396Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }) }
61512025-05-30T21:37:16.260ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 317.89953ms iteration = 12 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]}},"default_endpoint":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
61522025-05-30T21:37:16.262ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61532025-05-30T21:37:16.262ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61542025-05-30T21:37:16.272ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
61552025-05-30T21:37:16.272ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
61562025-05-30T21:37:16.272ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "b7d2644b-e3f1-4c52-83ae-685dc2a6becf", "date": "Fri, 30 May 2025 21:37:15 GMT"} })
61572025-05-30T21:37:16.274ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61582025-05-30T21:37:16.275ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 333.53948ms iteration = 23 status = {}
61592025-05-30T21:37:16.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
61602025-05-30T21:37:16.277ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
61612025-05-30T21:37:16.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
61622025-05-30T21:37:16.278ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 336.531314ms iteration = 23 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
61632025-05-30T21:37:16.278ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
61642025-05-30T21:37:16.278ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 0 version = 2
61652025-05-30T21:37:16.278ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
61662025-05-30T21:37:16.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
61672025-05-30T21:37:16.281ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
61682025-05-30T21:37:16.281ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
61692025-05-30T21:37:16.281ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-30 21:26:17.226115 UTC dns_group = internal generation = 1
61702025-05-30T21:37:16.281ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 339.519349ms iteration = 12 status = {"generation":1}
61712025-05-30T21:37:16.283ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61722025-05-30T21:37:16.285ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
61732025-05-30T21:37:16.286ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "236a74c8-92b6-4ce9-920e-f7c75000da0d", "date": "Fri, 30 May 2025 21:37:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
61742025-05-30T21:37:16.286ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
61752025-05-30T21:37:16.288ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "3ead8b1f-c996-49eb-ba97-9c59f5a2d87d", "date": "Fri, 30 May 2025 21:37:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
61762025-05-30T21:37:16.288ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
61772025-05-30T21:37:16.288ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
61782025-05-30T21:37:16.288ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
61792025-05-30T21:37:16.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
61802025-05-30T21:37:16.290ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "64b71094-1df2-4e26-a240-3a3999770f5e", "date": "Fri, 30 May 2025 21:37:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
61812025-05-30T21:37:16.290ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
61822025-05-30T21:37:16.292ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
61832025-05-30T21:37:16.292ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 349.717996ms iteration = 14 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
61842025-05-30T21:37:16.293ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 1 version = 2
61852025-05-30T21:37:16.293ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
61862025-05-30T21:37:16.293ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-30 21:26:18.957109 UTC dns_group = external generation = 2
61872025-05-30T21:37:16.293ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 351.594557ms iteration = 12 status = {"generation":2}
61882025-05-30T21:37:16.308ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
61892025-05-30T21:37:16.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61902025-05-30T21:37:16.310ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61912025-05-30T21:37:16.312ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
61922025-05-30T21:37:16.312ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
61932025-05-30T21:37:16.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61942025-05-30T21:37:16.326ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 384.428504ms iteration = 25 status = {"physical_disks_added":0}
61952025-05-30T21:37:16.346ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61962025-05-30T21:37:16.346ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61972025-05-30T21:37:16.349ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
61982025-05-30T21:37:16.349ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61992025-05-30T21:37:16.349ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62002025-05-30T21:37:16.356ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62012025-05-30T21:37:16.358ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
62022025-05-30T21:37:16.358ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
62032025-05-30T21:37:16.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
62042025-05-30T21:37:16.361ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
62052025-05-30T21:37:16.361ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 419.465467ms iteration = 12 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
62062025-05-30T21:37:16.361ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
62072025-05-30T21:37:16.361ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
62082025-05-30T21:37:16.362ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
62092025-05-30T21:37:16.362ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
62102025-05-30T21:37:16.362ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
62112025-05-30T21:37:16.362ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
62122025-05-30T21:37:16.362ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
62132025-05-30T21:37:16.362ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
62142025-05-30T21:37:16.362ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
62152025-05-30T21:37:16.362ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
62162025-05-30T21:37:16.362ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
62172025-05-30T21:37:16.362ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62182025-05-30T21:37:16.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62192025-05-30T21:37:16.364ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
62202025-05-30T21:37:16.364ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
62212025-05-30T21:37:16.462ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
62222025-05-30T21:37:16.481ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "b10f58d7-99e8-4063-a998-1a6e001a4da2", "date": "Fri, 30 May 2025 21:37:15 GMT"} })
62232025-05-30T21:37:16.481ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
62242025-05-30T21:37:16.481ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
62252025-05-30T21:37:16.481ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "159f4857-165f-456a-bfe9-48f5eca6584b", "content-length": "691", "date": "Fri, 30 May 2025 21:37:15 GMT"} })
62262025-05-30T21:37:16.481ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
62272025-05-30T21:37:16.481ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
62282025-05-30T21:37:16.481ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
62292025-05-30T21:37:16.481ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c668b01-aa66-4be3-ab23-fab37a310e83", "content-length": "682", "date": "Fri, 30 May 2025 21:37:15 GMT"} })
62302025-05-30T21:37:16.482ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62312025-05-30T21:37:16.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62322025-05-30T21:37:16.484ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 542.048415ms iteration = 23 status = {}
62332025-05-30T21:37:16.486ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62342025-05-30T21:37:16.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62352025-05-30T21:37:16.489ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62362025-05-30T21:37:16.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62372025-05-30T21:37:16.492ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62382025-05-30T21:37:16.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62392025-05-30T21:37:16.499ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 557.27559ms iteration = 23 status = {}
62402025-05-30T21:37:16.665ZDEBGnexus (producer-registration-task): registering as oximeter producer with Nexus registration_count = 4
62412025-05-30T21:37:16.665ZDEBGnexus (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221
62422025-05-30T21:37:16.694ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
62432025-05-30T21:37:16.695ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:36148
62442025-05-30T21:37:16.695ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:36148 req_id = 5e8e5675-0edc-4f17-a205-b0488633c4c6 roles = RoleSet { roles: {} } uri = /metrics/producers
62452025-05-30T21:37:16.695ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:36148 req_id = 5e8e5675-0edc-4f17-a205-b0488633c4c6 resource = Database result = Ok(()) uri = /metrics/producers
62462025-05-30T21:37:16.736ZINFOnexus: assigned collector to new producer collector_id = bc707af4-d9a3-402f-b2b6-642b29342fa8 file = nexus/src/app/oximeter.rs:76 producer_id = bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
62472025-05-30T21:37:16.736ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 41659 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:36148 req_id = 5e8e5675-0edc-4f17-a205-b0488633c4c6 response_code = 201 uri = /metrics/producers
62482025-05-30T21:37:16.736ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "5e8e5675-0edc-4f17-a205-b0488633c4c6", "content-length": "41", "date": "Fri, 30 May 2025 21:37:16 GMT"} })
62492025-05-30T21:37:16.737ZDEBGnexus (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
62502025-05-30T21:37:18.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 313 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 19a1d098-d602-404d-8061-9743b15e246e response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
62512025-05-30T21:37:25.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 68 reason = Timeout
62522025-05-30T21:37:25.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62532025-05-30T21:37:25.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 68 reason = Timeout
62542025-05-30T21:37:25.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62552025-05-30T21:37:25.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62562025-05-30T21:37:25.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62572025-05-30T21:37:25.978ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62582025-05-30T21:37:25.979ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62592025-05-30T21:37:25.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62602025-05-30T21:37:25.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
62612025-05-30T21:37:25.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62622025-05-30T21:37:25.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62632025-05-30T21:37:25.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62642025-05-30T21:37:25.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62652025-05-30T21:37:26.012ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 70.225147ms iteration = 68 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
62662025-05-30T21:37:26.043ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62672025-05-30T21:37:26.046ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
62682025-05-30T21:37:26.046ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62692025-05-30T21:37:26.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62702025-05-30T21:37:26.063ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
62712025-05-30T21:37:26.063ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 120.843978ms iteration = 68 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
62722025-05-30T21:37:28.768ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:36225
62732025-05-30T21:37:28.768ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36225 req_id = 8c5b7c83-3eb5-4efe-972d-6ce68a3814ee roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
62742025-05-30T21:37:28.768ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36225 req_id = 8c5b7c83-3eb5-4efe-972d-6ce68a3814ee resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
62752025-05-30T21:37:28.773ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 4823 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36225 req_id = 8c5b7c83-3eb5-4efe-972d-6ce68a3814ee response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
62762025-05-30T21:37:28.773ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36225 req_id = c732002a-0096-4368-a491-252da7b1f89e roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
62772025-05-30T21:37:28.773ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36225 req_id = c732002a-0096-4368-a491-252da7b1f89e resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
62782025-05-30T21:37:28.776ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3281 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36225 req_id = c732002a-0096-4368-a491-252da7b1f89e response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
62792025-05-30T21:37:28.777ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 433 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = b3e97fed-0a58-499d-8d6a-87a2745f08a9 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
62802025-05-30T21:37:35.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
62812025-05-30T21:37:35.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
62822025-05-30T21:37:35.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62832025-05-30T21:37:35.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62842025-05-30T21:37:35.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62852025-05-30T21:37:35.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62862025-05-30T21:37:35.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62872025-05-30T21:37:35.949ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
62882025-05-30T21:37:35.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62892025-05-30T21:37:35.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62902025-05-30T21:37:35.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62912025-05-30T21:37:35.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62922025-05-30T21:37:35.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62932025-05-30T21:37:35.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
62942025-05-30T21:37:35.955ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62952025-05-30T21:37:35.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62962025-05-30T21:37:35.965ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
62972025-05-30T21:37:35.965ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.178194ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
62982025-05-30T21:37:35.979ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62992025-05-30T21:37:35.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63002025-05-30T21:37:36.011ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 68.337535ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
63012025-05-30T21:37:38.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 341 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = dcc03f93-ae3e-4f35-82d5-f5cd3b549ca4 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
63022025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
63032025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63042025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
63052025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 24 reason = Timeout
63062025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
63072025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
63082025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
63092025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
63102025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
63112025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
63122025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
63132025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
63142025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
63152025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
63162025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
63172025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
63182025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
63192025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
63202025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
63212025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
63222025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
63232025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
63242025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
63252025-05-30T21:37:45.942ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
63262025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
63272025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
63282025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
63292025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63302025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63312025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
63322025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
63332025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
63342025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 24 reason = Timeout
63352025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
63362025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
63372025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
63382025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63392025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
63402025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
63412025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
63422025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63432025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
63442025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
63452025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
63462025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
63472025-05-30T21:37:45.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
63482025-05-30T21:37:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
63492025-05-30T21:37:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63502025-05-30T21:37:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
63512025-05-30T21:37:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
63522025-05-30T21:37:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
63532025-05-30T21:37:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63542025-05-30T21:37:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
63552025-05-30T21:37:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
63562025-05-30T21:37:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
63572025-05-30T21:37:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63582025-05-30T21:37:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
63592025-05-30T21:37:45.945ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63602025-05-30T21:37:45.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63612025-05-30T21:37:45.946ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.876614ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
63622025-05-30T21:37:45.947ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63632025-05-30T21:37:45.948ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1800, tv_nsec: 883825771 } })
63642025-05-30T21:37:45.948ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 5.851848ms iteration = 24 status = {"errors":[],"requests_created_ok":[]}
63652025-05-30T21:37:45.948ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63662025-05-30T21:37:45.948ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
63672025-05-30T21:37:45.950ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
63682025-05-30T21:37:45.950ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
63692025-05-30T21:37:45.950ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
63702025-05-30T21:37:45.950ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
63712025-05-30T21:37:45.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
63722025-05-30T21:37:45.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
63732025-05-30T21:37:45.953ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1800, tv_nsec: 888740397 } })
63742025-05-30T21:37:45.953ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63752025-05-30T21:37:45.953ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
63762025-05-30T21:37:45.953ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63772025-05-30T21:37:45.956ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
63782025-05-30T21:37:45.956ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63792025-05-30T21:37:45.956ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
63802025-05-30T21:37:45.956ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
63812025-05-30T21:37:45.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63822025-05-30T21:37:45.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
63832025-05-30T21:37:45.958ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63842025-05-30T21:37:45.959ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
63852025-05-30T21:37:45.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
63862025-05-30T21:37:45.960ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
63872025-05-30T21:37:45.960ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63882025-05-30T21:37:45.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63892025-05-30T21:37:45.962ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63902025-05-30T21:37:45.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63912025-05-30T21:37:45.963ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63922025-05-30T21:37:45.967ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
63932025-05-30T21:37:45.967ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63942025-05-30T21:37:45.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63952025-05-30T21:37:45.987ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
63962025-05-30T21:37:45.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
63972025-05-30T21:37:45.989ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
63982025-05-30T21:37:45.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
63992025-05-30T21:37:45.990ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 48.237484ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
64002025-05-30T21:37:45.990ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
64012025-05-30T21:37:45.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
64022025-05-30T21:37:45.996ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64032025-05-30T21:37:45.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
64042025-05-30T21:37:45.999ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
64052025-05-30T21:37:45.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
64062025-05-30T21:37:46.002ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1800, tv_nsec: 938294396 } })
64072025-05-30T21:37:46.002ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
64082025-05-30T21:37:46.003ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 60.677977ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
64092025-05-30T21:37:46.005ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64102025-05-30T21:37:46.005ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
64112025-05-30T21:37:46.005ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64122025-05-30T21:37:46.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64132025-05-30T21:37:46.005ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 63.341637ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
64142025-05-30T21:37:46.006ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64152025-05-30T21:37:46.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64162025-05-30T21:37:46.006ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64172025-05-30T21:37:46.007ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 65.495462ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
64182025-05-30T21:37:46.007ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64192025-05-30T21:37:46.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64202025-05-30T21:37:46.008ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64212025-05-30T21:37:46.008ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
64222025-05-30T21:37:46.008ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64232025-05-30T21:37:46.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64242025-05-30T21:37:46.009ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
64252025-05-30T21:37:46.009ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64262025-05-30T21:37:46.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64272025-05-30T21:37:46.010ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64282025-05-30T21:37:46.010ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64292025-05-30T21:37:46.012ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64302025-05-30T21:37:46.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64312025-05-30T21:37:46.013ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
64322025-05-30T21:37:46.013ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64332025-05-30T21:37:46.013ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
64342025-05-30T21:37:46.013ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64352025-05-30T21:37:46.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64362025-05-30T21:37:46.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64372025-05-30T21:37:46.015ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64382025-05-30T21:37:46.018ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64392025-05-30T21:37:46.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64402025-05-30T21:37:46.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
64412025-05-30T21:37:46.021ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64422025-05-30T21:37:46.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64432025-05-30T21:37:46.022ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64442025-05-30T21:37:46.023ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 81.576617ms iteration = 24 status = 0
64452025-05-30T21:37:46.025ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
64462025-05-30T21:37:46.025ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64472025-05-30T21:37:46.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64482025-05-30T21:37:46.027ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
64492025-05-30T21:37:46.027ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 84.608462ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
64502025-05-30T21:37:46.035ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
64512025-05-30T21:37:46.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
64522025-05-30T21:37:46.039ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 96.766322ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
64532025-05-30T21:37:46.046ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64542025-05-30T21:37:46.046ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
64552025-05-30T21:37:46.046ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 104.029984ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
64562025-05-30T21:37:46.054ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
64572025-05-30T21:37:46.054ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 112.26503ms iteration = 24 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
64582025-05-30T21:37:46.076ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
64592025-05-30T21:37:46.076ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
64602025-05-30T21:37:46.076ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "04d23562-f7f2-4911-b971-e95b87916542", "content-length": "26", "date": "Fri, 30 May 2025 21:37:46 GMT"} })
64612025-05-30T21:37:46.076ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
64622025-05-30T21:37:46.078ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
64632025-05-30T21:37:46.088ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
64642025-05-30T21:37:46.089ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
64652025-05-30T21:37:46.116ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
64662025-05-30T21:37:46.116ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
64672025-05-30T21:37:46.116ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "311e143d-9563-4719-bcfe-1217fe904cf1", "content-length": "26", "date": "Fri, 30 May 2025 21:37:46 GMT"} })
64682025-05-30T21:37:46.116ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
64692025-05-30T21:37:46.118ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
64702025-05-30T21:37:46.142ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = instance_watcher body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vmms/402ec8d0-6aec-4146-ab73-e1be29274545/state
64712025-05-30T21:37:46.142ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = instance_watcher result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vmms/402ec8d0-6aec-4146-ab73-e1be29274545/state", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4ae329a-558a-498d-a84e-ed03fe6a9691", "content-length": "130", "date": "Fri, 30 May 2025 21:37:45 GMT"} })
64722025-05-30T21:37:46.145ZDEBGnexus (BackgroundTasks): updating instance state background_task = instance_watcher instance_id = 553760f6-3f2d-4147-8af1-be1c3ad12df1 sled_id = 5d926227-6a52-4938-b74d-21ba39bce2f9 state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-05-30T21:37:34.102182212Z }, migration_in: None, migration_out: None } vmm_id = 402ec8d0-6aec-4146-ab73-e1be29274545
64732025-05-30T21:37:46.145ZINFOnexus (BackgroundTasks): received new VMM runtime state from sled agent background_task = instance_watcher file = nexus/src/app/instance.rs:2140 instance_id = 553760f6-3f2d-4147-8af1-be1c3ad12df1 migration_state = Migrations { migration_in: None, migration_out: None } propolis_id = 402ec8d0-6aec-4146-ab73-e1be29274545 sled_id = 5d926227-6a52-4938-b74d-21ba39bce2f9 vmm_id = 402ec8d0-6aec-4146-ab73-e1be29274545 vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-05-30T21:37:34.102182212Z }
64742025-05-30T21:37:46.145ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 553760f6-3f2d-4147-8af1-be1c3ad12df1 roles = RoleSet { roles: {} } sled_id = 5d926227-6a52-4938-b74d-21ba39bce2f9 vmm_id = 402ec8d0-6aec-4146-ab73-e1be29274545
64752025-05-30T21:37:46.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 553760f6-3f2d-4147-8af1-be1c3ad12df1 resource = Database result = Ok(()) sled_id = 5d926227-6a52-4938-b74d-21ba39bce2f9 vmm_id = 402ec8d0-6aec-4146-ab73-e1be29274545
64762025-05-30T21:37:46.161ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
64772025-05-30T21:37:46.161ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
64782025-05-30T21:37:46.162ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9eb9cd24-6a76-4a46-a988-d91834e3dcab", "content-length": "26", "date": "Fri, 30 May 2025 21:37:46 GMT"} })
64792025-05-30T21:37:46.162ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
64802025-05-30T21:37:46.163ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
64812025-05-30T21:37:46.181ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 239.413398ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
64822025-05-30T21:37:46.191ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 249.371723ms iteration = 26 status = {"physical_disks_added":0}
64832025-05-30T21:37:46.192ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 1 total_failed = 0 total_incomplete = 0 total_instances = 1 update_sagas_queued = 0
64842025-05-30T21:37:46.195ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 252.677191ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{"running":1},"pruned_instances":0,"total_instances":1,"update_sagas_queued":0}
64852025-05-30T21:37:46.202ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 260.040046ms iteration = 24 status = {}
64862025-05-30T21:37:46.227ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
64872025-05-30T21:37:46.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
64882025-05-30T21:37:46.234ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
64892025-05-30T21:37:46.234ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
64902025-05-30T21:37:46.234ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "b767da01-726f-49eb-afdb-f15d3ecb13be", "content-length": "585", "date": "Fri, 30 May 2025 21:37:45 GMT"} })
64912025-05-30T21:37:46.234ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
64922025-05-30T21:37:46.234ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
64932025-05-30T21:37:46.236ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 294.505781ms iteration = 24 status = null
64942025-05-30T21:37:46.250ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
64952025-05-30T21:37:46.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
64962025-05-30T21:37:46.282ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {00131ff7-150c-4ffc-a853-62c3e3499084: {10.151.2.176: {8cf06590-884a-4473-ac6f-b3d617688cff}, 10.151.2.175: {8cf06590-884a-4473-ac6f-b3d617688cff}}} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
64972025-05-30T21:37:46.282ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
64982025-05-30T21:37:46.282ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "f0e7b447-a2ab-4f93-9293-f52685de710c", "date": "Fri, 30 May 2025 21:37:45 GMT"} })
64992025-05-30T21:37:46.282ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65002025-05-30T21:37:46.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65012025-05-30T21:37:46.283ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65022025-05-30T21:37:46.285ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65032025-05-30T21:37:46.285ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
65042025-05-30T21:37:46.285ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65052025-05-30T21:37:46.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65062025-05-30T21:37:46.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65072025-05-30T21:37:46.289ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65082025-05-30T21:37:46.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65092025-05-30T21:37:46.292ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65102025-05-30T21:37:46.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65112025-05-30T21:37:46.294ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65122025-05-30T21:37:46.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65132025-05-30T21:37:46.297ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65142025-05-30T21:37:46.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65152025-05-30T21:37:46.300ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65162025-05-30T21:37:46.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65172025-05-30T21:37:46.301ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
65182025-05-30T21:37:46.301ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
65192025-05-30T21:37:46.301ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
65202025-05-30T21:37:46.301ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "67de1241-7308-4996-bec0-5de98d7f4fa5", "date": "Fri, 30 May 2025 21:37:45 GMT"} })
65212025-05-30T21:37:46.302ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
65222025-05-30T21:37:46.302ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65232025-05-30T21:37:46.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65242025-05-30T21:37:46.303ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 360.958605ms iteration = 24 status = {}
65252025-05-30T21:37:46.314ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
65262025-05-30T21:37:46.314ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
65272025-05-30T21:37:46.314ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
65282025-05-30T21:37:46.314ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
65292025-05-30T21:37:46.314ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
65302025-05-30T21:37:46.314ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
65312025-05-30T21:37:46.314ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
65322025-05-30T21:37:46.314ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
65332025-05-30T21:37:46.314ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
65342025-05-30T21:37:46.314ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
65352025-05-30T21:37:46.314ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
65362025-05-30T21:37:46.314ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65372025-05-30T21:37:46.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65382025-05-30T21:37:46.317ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
65392025-05-30T21:37:46.317ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
65402025-05-30T21:37:46.335ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65412025-05-30T21:37:46.338ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
65422025-05-30T21:37:46.338ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65432025-05-30T21:37:46.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65442025-05-30T21:37:46.426ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "d1de8570-b946-4dc7-b76a-a62568e8950a", "date": "Fri, 30 May 2025 21:37:46 GMT"} })
65452025-05-30T21:37:46.426ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
65462025-05-30T21:37:46.426ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
65472025-05-30T21:37:46.427ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "b086381f-c310-4f5d-b521-806c8a752a42", "content-length": "682", "date": "Fri, 30 May 2025 21:37:46 GMT"} })
65482025-05-30T21:37:46.427ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65492025-05-30T21:37:46.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65502025-05-30T21:37:46.431ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65512025-05-30T21:37:46.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65522025-05-30T21:37:46.434ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65532025-05-30T21:37:46.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65542025-05-30T21:37:46.438ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65552025-05-30T21:37:46.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65562025-05-30T21:37:46.444ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
65572025-05-30T21:37:46.445ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "041b0e5d-d6b7-4c0b-8092-1126d9ee7d88", "content-length": "811", "date": "Fri, 30 May 2025 21:37:45 GMT"} })
65582025-05-30T21:37:46.445ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {VirtualNetworkInterfaceHost { physical_host_ip: fd00:1122:3344:101::1, virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), vni: Vni(14103399) }} sled = sock
65592025-05-30T21:37:46.445ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
65602025-05-30T21:37:46.445ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
65612025-05-30T21:37:46.447ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 504.798567ms iteration = 24 status = {}
65622025-05-30T21:37:46.447ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 505.118711ms iteration = 24 status = {}
65632025-05-30T21:37:48.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 435 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 41b5b896-734b-408a-b790-3acf213f29ee response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
65642025-05-30T21:37:55.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
65652025-05-30T21:37:55.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
65662025-05-30T21:37:55.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
65672025-05-30T21:37:55.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65682025-05-30T21:37:55.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65692025-05-30T21:37:55.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
65702025-05-30T21:37:55.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65712025-05-30T21:37:55.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
65722025-05-30T21:37:55.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65732025-05-30T21:37:55.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65742025-05-30T21:37:55.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65752025-05-30T21:37:55.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65762025-05-30T21:37:55.979ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
65772025-05-30T21:37:55.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
65782025-05-30T21:37:55.981ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 39.232491ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
65792025-05-30T21:37:55.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65802025-05-30T21:37:55.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
65812025-05-30T21:37:55.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65822025-05-30T21:37:55.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65832025-05-30T21:37:56.003ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
65842025-05-30T21:37:56.003ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 61.518007ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
65852025-05-30T21:37:58.768ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36225 req_id = ca8d1a74-3f87-4c9e-8429-1d3d788b8a5e roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
65862025-05-30T21:37:58.768ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36225 req_id = ca8d1a74-3f87-4c9e-8429-1d3d788b8a5e resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
65872025-05-30T21:37:58.771ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3266 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36225 req_id = ca8d1a74-3f87-4c9e-8429-1d3d788b8a5e response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
65882025-05-30T21:37:58.771ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36225 req_id = 17ca5980-d77a-49d9-a386-5a53287ff31c roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
65892025-05-30T21:37:58.771ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36225 req_id = 17ca5980-d77a-49d9-a386-5a53287ff31c resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
65902025-05-30T21:37:58.774ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2659 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36225 req_id = 17ca5980-d77a-49d9-a386-5a53287ff31c response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
65912025-05-30T21:37:58.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 363 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = c94acd25-8dc0-4f39-a098-c9eb8f27f409 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
65922025-05-30T21:38:05.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 72 reason = Timeout
65932025-05-30T21:38:05.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 72 reason = Timeout
65942025-05-30T21:38:05.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65952025-05-30T21:38:05.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
65962025-05-30T21:38:05.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
65972025-05-30T21:38:05.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65982025-05-30T21:38:05.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65992025-05-30T21:38:05.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
66002025-05-30T21:38:05.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66012025-05-30T21:38:05.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66022025-05-30T21:38:05.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66032025-05-30T21:38:05.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66042025-05-30T21:38:05.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66052025-05-30T21:38:05.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66062025-05-30T21:38:05.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66072025-05-30T21:38:05.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
66082025-05-30T21:38:05.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66092025-05-30T21:38:05.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66102025-05-30T21:38:05.999ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
66112025-05-30T21:38:05.999ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 56.865023ms iteration = 72 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
66122025-05-30T21:38:06.013ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 70.808313ms iteration = 72 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
66132025-05-30T21:38:08.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 353 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = efa0f42a-1565-4d2a-971c-34ea9c11fe17 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
66142025-05-30T21:38:15.941ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 13 reason = Timeout
66152025-05-30T21:38:15.941ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
66162025-05-30T21:38:15.941ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
66172025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
66182025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 73 reason = Timeout
66192025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66202025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 25 reason = Timeout
66212025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 25 reason = Timeout
66222025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
66232025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
66242025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 25 reason = Timeout
66252025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
66262025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 25 reason = Timeout
66272025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
66282025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
66292025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 25 reason = Timeout
66302025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
66312025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
66322025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 25 reason = Timeout
66332025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
66342025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 25 reason = Timeout
66352025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
66362025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 25 reason = Timeout
66372025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
66382025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 25 reason = Timeout
66392025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
66402025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66412025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 25 reason = Timeout
66422025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
66432025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 25 reason = Timeout
66442025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 25 reason = Timeout
66452025-05-30T21:38:15.942ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
66462025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66472025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
66482025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 13 reason = Timeout
66492025-05-30T21:38:15.942ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-30 21:28:15.942778560 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
66502025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
66512025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 25 reason = Timeout
66522025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
66532025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 27 reason = Timeout
66542025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
66552025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 25 reason = Timeout
66562025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66572025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 13 reason = Timeout
66582025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 73 reason = Timeout
66592025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 13 reason = Timeout
66602025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
66612025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 14 reason = Timeout
66622025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66632025-05-30T21:38:15.942ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e
66642025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.921\u{b5}s iteration = 14 status = {"enabled":false,"target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e"}
66652025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 13 reason = Timeout
66662025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 6 reason = Timeout
66672025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
66682025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 13 reason = Timeout
66692025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 13 reason = Timeout
66702025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
66712025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
66722025-05-30T21:38:15.942ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 13 reason = Timeout
66732025-05-30T21:38:15.943ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
66742025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 13 reason = Timeout
66752025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
66762025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
66772025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 13 reason = Timeout
66782025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
66792025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
66802025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 13 reason = Timeout
66812025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
66822025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 15 reason = Timeout
66832025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 13 reason = Timeout
66842025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
66852025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 25 reason = Timeout
66862025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
66872025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 15 reason = Timeout
66882025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 25 reason = Timeout
66892025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
66902025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
66912025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
66922025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
66932025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
66942025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
66952025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
66962025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66972025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
66982025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
66992025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
67002025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
67012025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
67022025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
67032025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
67042025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
67052025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
67062025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67072025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
67082025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
67092025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
67102025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
67112025-05-30T21:38:15.943ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2a161087-f464-41f8-b020-0f9e83fd5351.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1830, tv_nsec: 879011719 } })
67122025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
67132025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
67142025-05-30T21:38:15.943ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0413f6b4-86a1-42a1-919a-e8f42ff9f638.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("acb1452d-0a02-4b36-a462-c8ce2f96a488.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:2::1))) }], valid_until: Instant { tv_sec: 1830, tv_nsec: 879034369 } })
67152025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
67162025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
67172025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
67182025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
67192025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
67202025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
67212025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
67222025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
67232025-05-30T21:38:15.943ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
67242025-05-30T21:38:15.944ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
67252025-05-30T21:38:15.944ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } dns_group = internal
67262025-05-30T21:38:15.944ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.168613ms iteration = 13 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
67272025-05-30T21:38:15.944ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.252234ms iteration = 13 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353"]}
67282025-05-30T21:38:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
67292025-05-30T21:38:15.944ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
67302025-05-30T21:38:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
67312025-05-30T21:38:15.946ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.926703ms iteration = 25 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
67322025-05-30T21:38:15.947ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1830, tv_nsec: 882906643 } })
67332025-05-30T21:38:15.947ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
67342025-05-30T21:38:15.948ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67352025-05-30T21:38:15.950ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
67362025-05-30T21:38:15.950ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
67372025-05-30T21:38:15.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
67382025-05-30T21:38:15.952ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67392025-05-30T21:38:15.953ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
67402025-05-30T21:38:15.953ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
67412025-05-30T21:38:15.953ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
67422025-05-30T21:38:15.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
67432025-05-30T21:38:15.954ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
67442025-05-30T21:38:15.954ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
67452025-05-30T21:38:15.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
67462025-05-30T21:38:15.956ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1830, tv_nsec: 892088269 } })
67472025-05-30T21:38:15.956ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
67482025-05-30T21:38:15.957ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67492025-05-30T21:38:15.958ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67502025-05-30T21:38:15.959ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 16.592531ms iteration = 13 status = {"errors":[],"found_port_ok":[]}
67512025-05-30T21:38:15.960ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
67522025-05-30T21:38:15.960ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67532025-05-30T21:38:15.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67542025-05-30T21:38:15.960ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
67552025-05-30T21:38:15.960ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
67562025-05-30T21:38:15.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
67572025-05-30T21:38:15.961ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
67582025-05-30T21:38:15.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
67592025-05-30T21:38:15.967ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:202 num_receivers = 0
67602025-05-30T21:38:15.967ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 24.621153ms iteration = 13 status = {"by_rx":{},"error":null}
67612025-05-30T21:38:15.968ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-30 21:28:15.942778560 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
67622025-05-30T21:38:15.968ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 25.815847ms iteration = 13 status = {"expiration":"2025-05-30T21:28:15.942778560Z","pruned":[]}
67632025-05-30T21:38:15.982ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 40.261053ms iteration = 25 status = {"errors":[],"finish_invoked_ok":[]}
67642025-05-30T21:38:15.995ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67652025-05-30T21:38:15.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67662025-05-30T21:38:15.998ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67672025-05-30T21:38:16.000ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
67682025-05-30T21:38:16.000ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 57.575561ms iteration = 25 status = {"errors":[],"garbage_collect_requested":[]}
67692025-05-30T21:38:16.000ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67702025-05-30T21:38:16.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
67712025-05-30T21:38:16.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
67722025-05-30T21:38:16.002ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
67732025-05-30T21:38:16.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
67742025-05-30T21:38:16.004ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67752025-05-30T21:38:16.004ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
67762025-05-30T21:38:16.005ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67772025-05-30T21:38:16.005ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67782025-05-30T21:38:16.005ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 63.126695ms iteration = 25 status = {"errors":[],"requests_created_ok":[]}
67792025-05-30T21:38:16.007ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67802025-05-30T21:38:16.008ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1830, tv_nsec: 943541490 } })
67812025-05-30T21:38:16.008ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
67822025-05-30T21:38:16.008ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
67832025-05-30T21:38:16.008ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
67842025-05-30T21:38:16.008ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
67852025-05-30T21:38:16.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
67862025-05-30T21:38:16.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
67872025-05-30T21:38:16.008ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
67882025-05-30T21:38:16.008ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
67892025-05-30T21:38:16.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
67902025-05-30T21:38:16.009ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67912025-05-30T21:38:16.009ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
67922025-05-30T21:38:16.009ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
67932025-05-30T21:38:16.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
67942025-05-30T21:38:16.010ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
67952025-05-30T21:38:16.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
67962025-05-30T21:38:16.011ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
67972025-05-30T21:38:16.011ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
67982025-05-30T21:38:16.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
67992025-05-30T21:38:16.012ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
68002025-05-30T21:38:16.012ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
68012025-05-30T21:38:16.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68022025-05-30T21:38:16.014ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
68032025-05-30T21:38:16.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
68042025-05-30T21:38:16.015ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
68052025-05-30T21:38:16.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68062025-05-30T21:38:16.015ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68072025-05-30T21:38:16.017ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68082025-05-30T21:38:16.017ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
68092025-05-30T21:38:16.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
68102025-05-30T21:38:16.018ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68112025-05-30T21:38:16.019ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
68122025-05-30T21:38:16.020ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
68132025-05-30T21:38:16.020ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68142025-05-30T21:38:16.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68152025-05-30T21:38:16.021ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68162025-05-30T21:38:16.022ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
68172025-05-30T21:38:16.022ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
68182025-05-30T21:38:16.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
68192025-05-30T21:38:16.022ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
68202025-05-30T21:38:16.022ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
68212025-05-30T21:38:16.022ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 79.898738ms iteration = 13 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
68222025-05-30T21:38:16.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
68232025-05-30T21:38:16.024ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
68242025-05-30T21:38:16.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
68252025-05-30T21:38:16.027ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 85.198358ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
68262025-05-30T21:38:16.029ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68272025-05-30T21:38:16.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68282025-05-30T21:38:16.029ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68292025-05-30T21:38:16.030ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 87.444164ms iteration = 13 status = {"dispatched":[],"errors":[],"glob_version":"144.0.0","globs_reprocessed":{},"no_receivers":[]}
68302025-05-30T21:38:16.031ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68312025-05-30T21:38:16.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68322025-05-30T21:38:16.031ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 1 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
68332025-05-30T21:38:16.031ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 89.193525ms iteration = 25 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":1,"total_instances":0,"update_sagas_queued":0}
68342025-05-30T21:38:16.032ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68352025-05-30T21:38:16.034ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
68362025-05-30T21:38:16.034ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
68372025-05-30T21:38:16.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
68382025-05-30T21:38:16.035ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
68392025-05-30T21:38:16.035ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
68402025-05-30T21:38:16.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
68412025-05-30T21:38:16.037ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68422025-05-30T21:38:16.040ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
68432025-05-30T21:38:16.041ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68442025-05-30T21:38:16.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68452025-05-30T21:38:16.050ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
68462025-05-30T21:38:16.050ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
68472025-05-30T21:38:16.050ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
68482025-05-30T21:38:16.050ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
68492025-05-30T21:38:16.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
68502025-05-30T21:38:16.051ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
68512025-05-30T21:38:16.065ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68522025-05-30T21:38:16.070ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
68532025-05-30T21:38:16.070ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
68542025-05-30T21:38:16.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
68552025-05-30T21:38:16.074ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
68562025-05-30T21:38:16.074ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
68572025-05-30T21:38:16.074ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
68582025-05-30T21:38:16.074ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68592025-05-30T21:38:16.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
68602025-05-30T21:38:16.076ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
68612025-05-30T21:38:16.076ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
68622025-05-30T21:38:16.076ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
68632025-05-30T21:38:16.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68642025-05-30T21:38:16.077ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68652025-05-30T21:38:16.078ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
68662025-05-30T21:38:16.078ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
68672025-05-30T21:38:16.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
68682025-05-30T21:38:16.079ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
68692025-05-30T21:38:16.079ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
68702025-05-30T21:38:16.079ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
68712025-05-30T21:38:16.079ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
68722025-05-30T21:38:16.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
68732025-05-30T21:38:16.081ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
68742025-05-30T21:38:16.081ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
68752025-05-30T21:38:16.082ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 139.425681ms iteration = 73 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
68762025-05-30T21:38:16.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
68772025-05-30T21:38:16.084ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 141.479005ms iteration = 6 status = {"nsuccess":5}
68782025-05-30T21:38:16.084ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
68792025-05-30T21:38:16.085ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68802025-05-30T21:38:16.087ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
68812025-05-30T21:38:16.087ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
68822025-05-30T21:38:16.087ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-30 21:26:17.226115 UTC dns_group = internal generation = 1
68832025-05-30T21:38:16.087ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 146.000516ms iteration = 13 status = {"generation":1}
68842025-05-30T21:38:16.088ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
68852025-05-30T21:38:16.089ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68862025-05-30T21:38:16.091ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
68872025-05-30T21:38:16.091ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
68882025-05-30T21:38:16.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
68892025-05-30T21:38:16.091ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68902025-05-30T21:38:16.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68912025-05-30T21:38:16.092ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
68922025-05-30T21:38:16.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68932025-05-30T21:38:16.095ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68942025-05-30T21:38:16.095ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68952025-05-30T21:38:16.095ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68962025-05-30T21:38:16.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68972025-05-30T21:38:16.096ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
68982025-05-30T21:38:16.096ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
68992025-05-30T21:38:16.098ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "63a909d1-3bfa-4308-985a-6d5038344f4a", "content-length": "26", "date": "Fri, 30 May 2025 21:38:16 GMT"} })
69002025-05-30T21:38:16.098ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
69012025-05-30T21:38:16.099ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
69022025-05-30T21:38:16.099ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
69032025-05-30T21:38:16.099ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
69042025-05-30T21:38:16.099ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
69052025-05-30T21:38:16.100ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
69062025-05-30T21:38:16.100ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69072025-05-30T21:38:16.100ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69082025-05-30T21:38:16.102ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69092025-05-30T21:38:16.102ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 159.840735ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
69102025-05-30T21:38:16.104ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
69112025-05-30T21:38:16.104ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
69122025-05-30T21:38:16.104ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 161.12188ms iteration = 13 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
69132025-05-30T21:38:16.106ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
69142025-05-30T21:38:16.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
69152025-05-30T21:38:16.107ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
69162025-05-30T21:38:16.107ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69172025-05-30T21:38:16.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
69182025-05-30T21:38:16.112ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 170.065422ms iteration = 25 status = 0
69192025-05-30T21:38:16.116ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
69202025-05-30T21:38:16.116ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 173.837986ms iteration = 73 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
69212025-05-30T21:38:16.120ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69222025-05-30T21:38:16.122ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
69232025-05-30T21:38:16.122ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
69242025-05-30T21:38:16.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
69252025-05-30T21:38:16.143ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
69262025-05-30T21:38:16.143ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
69272025-05-30T21:38:16.143ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
69282025-05-30T21:38:16.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
69292025-05-30T21:38:16.143ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa37b05f-96c5-4e2c-838f-1fad3cf609d1", "content-length": "26", "date": "Fri, 30 May 2025 21:38:16 GMT"} })
69302025-05-30T21:38:16.143ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
69312025-05-30T21:38:16.146ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
69322025-05-30T21:38:16.147ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
69332025-05-30T21:38:16.147ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
69342025-05-30T21:38:16.147ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 204.622419ms iteration = 25 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
69352025-05-30T21:38:16.156ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
69362025-05-30T21:38:16.156ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 0 version = 2
69372025-05-30T21:38:16.156ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
69382025-05-30T21:38:16.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
69392025-05-30T21:38:16.158ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 215.9267ms iteration = 25 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
69402025-05-30T21:38:16.161ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69412025-05-30T21:38:16.162ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
69422025-05-30T21:38:16.162ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
69432025-05-30T21:38:16.165ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
69442025-05-30T21:38:16.165ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 222.720627ms iteration = 25 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
69452025-05-30T21:38:16.166ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
69462025-05-30T21:38:16.166ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
69472025-05-30T21:38:16.166ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
69482025-05-30T21:38:16.187ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-30T21:26:13.986396Z, time_modified: 2025-05-30T21:26:13.986396Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
69492025-05-30T21:38:16.187ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 244.974673ms iteration = 13 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]}},"default_endpoint":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
69502025-05-30T21:38:16.241ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
69512025-05-30T21:38:16.242ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "bc6b1175-dfba-409b-9206-219e83888484", "date": "Fri, 30 May 2025 21:38:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
69522025-05-30T21:38:16.242ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
69532025-05-30T21:38:16.243ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "23f1e86e-176a-41a1-8b22-686dfd1e365f", "date": "Fri, 30 May 2025 21:38:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
69542025-05-30T21:38:16.243ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
69552025-05-30T21:38:16.245ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
69562025-05-30T21:38:16.245ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 302.387742ms iteration = 15 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
69572025-05-30T21:38:16.256ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
69582025-05-30T21:38:16.256ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
69592025-05-30T21:38:16.256ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9255c0e-05b8-41e3-9571-78882fb48a3a", "content-length": "26", "date": "Fri, 30 May 2025 21:38:16 GMT"} })
69602025-05-30T21:38:16.256ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
69612025-05-30T21:38:16.258ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
69622025-05-30T21:38:16.260ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 318.034322ms iteration = 27 status = {"physical_disks_added":0}
69632025-05-30T21:38:16.271ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
69642025-05-30T21:38:16.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
69652025-05-30T21:38:16.316ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
69662025-05-30T21:38:16.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
69672025-05-30T21:38:16.330ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
69682025-05-30T21:38:16.330ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 387.333128ms iteration = 13 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
69692025-05-30T21:38:16.343ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 400.213986ms iteration = 25 status = {}
69702025-05-30T21:38:16.360ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
69712025-05-30T21:38:16.394ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
69722025-05-30T21:38:16.394ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
69732025-05-30T21:38:16.394ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "01f2e166-99c9-4b0a-b8b7-7896fd610377", "content-length": "372", "date": "Fri, 30 May 2025 21:38:15 GMT"} })
69742025-05-30T21:38:16.394ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69752025-05-30T21:38:16.395ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69762025-05-30T21:38:16.396ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 1 version = 2
69772025-05-30T21:38:16.396ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
69782025-05-30T21:38:16.396ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-30 21:26:18.957109 UTC dns_group = external generation = 2
69792025-05-30T21:38:16.396ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 453.751561ms iteration = 13 status = {"generation":2}
69802025-05-30T21:38:16.397ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69812025-05-30T21:38:16.397ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 454.864803ms iteration = 25 status = null
69822025-05-30T21:38:16.399ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
69832025-05-30T21:38:16.399ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
69842025-05-30T21:38:16.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
69852025-05-30T21:38:16.400ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69862025-05-30T21:38:16.402ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
69872025-05-30T21:38:16.402ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
69882025-05-30T21:38:16.402ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
69892025-05-30T21:38:16.443ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
69902025-05-30T21:38:16.443ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
69912025-05-30T21:38:16.443ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "7bb7b84b-3afc-437a-a911-e0c36befed96", "date": "Fri, 30 May 2025 21:38:15 GMT"} })
69922025-05-30T21:38:16.443ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69932025-05-30T21:38:16.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69942025-05-30T21:38:16.445ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69952025-05-30T21:38:16.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69962025-05-30T21:38:16.448ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69972025-05-30T21:38:16.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69982025-05-30T21:38:16.456ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
69992025-05-30T21:38:16.456ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "b4e5ec19-43d4-4ec1-a18e-c616d6684cfc", "date": "Fri, 30 May 2025 21:38:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
70002025-05-30T21:38:16.456ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
70012025-05-30T21:38:16.459ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "810a481b-386f-4715-a4ee-065d9b269947", "date": "Fri, 30 May 2025 21:38:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
70022025-05-30T21:38:16.459ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
70032025-05-30T21:38:16.461ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "378e800f-29f3-4c57-8367-aef84f6e4663", "date": "Fri, 30 May 2025 21:38:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
70042025-05-30T21:38:16.461ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
70052025-05-30T21:38:16.463ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
70062025-05-30T21:38:16.463ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 520.814881ms iteration = 15 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
70072025-05-30T21:38:16.473ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
70082025-05-30T21:38:16.473ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "5772fd00-877b-4c9b-b087-ec74fb183f4b", "content-length": "691", "date": "Fri, 30 May 2025 21:38:15 GMT"} })
70092025-05-30T21:38:16.473ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
70102025-05-30T21:38:16.473ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
70112025-05-30T21:38:16.473ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
70122025-05-30T21:38:16.475ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 531.959769ms iteration = 25 status = {}
70132025-05-30T21:38:16.475ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
70142025-05-30T21:38:16.475ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
70152025-05-30T21:38:16.476ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "15c7bdd7-05d0-4099-bc8a-06e25d1ab300", "date": "Fri, 30 May 2025 21:38:15 GMT"} })
70162025-05-30T21:38:16.478ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 535.52325ms iteration = 25 status = {}
70172025-05-30T21:38:16.487ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70182025-05-30T21:38:16.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70192025-05-30T21:38:16.489ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
70202025-05-30T21:38:16.489ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70212025-05-30T21:38:16.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70222025-05-30T21:38:16.501ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
70232025-05-30T21:38:16.501ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
70242025-05-30T21:38:16.502ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
70252025-05-30T21:38:16.502ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
70262025-05-30T21:38:16.502ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
70272025-05-30T21:38:16.502ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
70282025-05-30T21:38:16.502ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
70292025-05-30T21:38:16.502ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
70302025-05-30T21:38:16.502ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
70312025-05-30T21:38:16.502ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
70322025-05-30T21:38:16.502ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
70332025-05-30T21:38:16.502ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70342025-05-30T21:38:16.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70352025-05-30T21:38:16.504ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
70362025-05-30T21:38:16.504ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
70372025-05-30T21:38:16.610ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "56489f48-2e07-43b2-ba15-1b70351b5fe0", "date": "Fri, 30 May 2025 21:38:15 GMT"} })
70382025-05-30T21:38:16.610ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
70392025-05-30T21:38:16.610ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
70402025-05-30T21:38:16.610ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "90c2f22f-8b5a-4d24-9455-d65673f58c94", "content-length": "682", "date": "Fri, 30 May 2025 21:38:15 GMT"} })
70412025-05-30T21:38:16.610ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70422025-05-30T21:38:16.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70432025-05-30T21:38:16.614ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70442025-05-30T21:38:16.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70452025-05-30T21:38:16.618ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70462025-05-30T21:38:16.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70472025-05-30T21:38:16.621ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70482025-05-30T21:38:16.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70492025-05-30T21:38:16.627ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 684.991257ms iteration = 25 status = {}
70502025-05-30T21:38:18.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 329 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 889f4ea2-8b5e-46d6-9cd2-f9cb60a27e68 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
70512025-05-30T21:38:25.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 74 reason = Timeout
70522025-05-30T21:38:25.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70532025-05-30T21:38:25.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 74 reason = Timeout
70542025-05-30T21:38:25.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
70552025-05-30T21:38:25.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70562025-05-30T21:38:25.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
70572025-05-30T21:38:25.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70582025-05-30T21:38:25.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
70592025-05-30T21:38:25.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70602025-05-30T21:38:25.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70612025-05-30T21:38:25.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70622025-05-30T21:38:25.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70632025-05-30T21:38:25.976ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
70642025-05-30T21:38:25.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
70652025-05-30T21:38:25.978ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 35.305065ms iteration = 74 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
70662025-05-30T21:38:25.982ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70672025-05-30T21:38:25.984ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
70682025-05-30T21:38:25.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70692025-05-30T21:38:25.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70702025-05-30T21:38:25.996ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
70712025-05-30T21:38:25.996ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 53.125671ms iteration = 74 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
70722025-05-30T21:38:28.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 387 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = a027f1cf-ddc9-4718-ae73-fe777f287652 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
70732025-05-30T21:38:35.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 75 reason = Timeout
70742025-05-30T21:38:35.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 75 reason = Timeout
70752025-05-30T21:38:35.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70762025-05-30T21:38:35.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
70772025-05-30T21:38:35.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70782025-05-30T21:38:35.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
70792025-05-30T21:38:35.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70802025-05-30T21:38:35.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
70812025-05-30T21:38:35.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70822025-05-30T21:38:35.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70832025-05-30T21:38:35.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70842025-05-30T21:38:35.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70852025-05-30T21:38:35.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70862025-05-30T21:38:35.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
70872025-05-30T21:38:35.955ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70882025-05-30T21:38:35.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70892025-05-30T21:38:35.963ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
70902025-05-30T21:38:35.963ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.671358ms iteration = 75 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
70912025-05-30T21:38:35.975ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
70922025-05-30T21:38:35.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
70932025-05-30T21:38:35.977ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.986882ms iteration = 75 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
70942025-05-30T21:38:38.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 304 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = beb9d8be-ffbc-4122-a9c0-838f1ccc9085 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
70952025-05-30T21:38:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 76 reason = Timeout
70962025-05-30T21:38:45.942ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
70972025-05-30T21:38:45.942ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 26 reason = Timeout
70982025-05-30T21:38:45.942ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
70992025-05-30T21:38:45.942ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
71002025-05-30T21:38:45.942ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 26 reason = Timeout
71012025-05-30T21:38:45.942ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
71022025-05-30T21:38:45.942ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 26 reason = Timeout
71032025-05-30T21:38:45.942ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 26 reason = Timeout
71042025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
71052025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
71062025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 26 reason = Timeout
71072025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
71082025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 26 reason = Timeout
71092025-05-30T21:38:45.943ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
71102025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 26 reason = Timeout
71112025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
71122025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71132025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 76 reason = Timeout
71142025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71152025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 26 reason = Timeout
71162025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
71172025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 26 reason = Timeout
71182025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 26 reason = Timeout
71192025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
71202025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
71212025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 28 reason = Timeout
71222025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
71232025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 26 reason = Timeout
71242025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
71252025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 26 reason = Timeout
71262025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
71272025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 26 reason = Timeout
71282025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
71292025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 26 reason = Timeout
71302025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
71312025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 26 reason = Timeout
71322025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
71332025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 26 reason = Timeout
71342025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71352025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
71362025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
71372025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
71382025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71392025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
71402025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
71412025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71422025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
71432025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
71442025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
71452025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
71462025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
71472025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
71482025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71492025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
71502025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
71512025-05-30T21:38:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
71522025-05-30T21:38:45.946ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.514511ms iteration = 26 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
71532025-05-30T21:38:45.947ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71542025-05-30T21:38:45.948ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 5.366562ms iteration = 26 status = {"errors":[],"garbage_collect_requested":[]}
71552025-05-30T21:38:45.949ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1860, tv_nsec: 884338063 } })
71562025-05-30T21:38:45.949ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
71572025-05-30T21:38:45.949ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71582025-05-30T21:38:45.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71592025-05-30T21:38:45.949ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
71602025-05-30T21:38:45.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71612025-05-30T21:38:45.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
71622025-05-30T21:38:45.955ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71632025-05-30T21:38:45.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71642025-05-30T21:38:45.955ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1860, tv_nsec: 891172232 } })
71652025-05-30T21:38:45.956ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71662025-05-30T21:38:45.956ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
71672025-05-30T21:38:45.956ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71682025-05-30T21:38:45.959ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
71692025-05-30T21:38:45.959ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
71702025-05-30T21:38:45.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
71712025-05-30T21:38:45.959ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
71722025-05-30T21:38:45.959ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
71732025-05-30T21:38:45.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71742025-05-30T21:38:45.962ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
71752025-05-30T21:38:45.962ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71762025-05-30T21:38:45.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71772025-05-30T21:38:45.963ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1860, tv_nsec: 898316914 } })
71782025-05-30T21:38:45.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71792025-05-30T21:38:45.963ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
71802025-05-30T21:38:45.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 44.788994ms iteration = 26 status = {"errors":[],"finish_invoked_ok":[]}
71812025-05-30T21:38:45.992ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
71822025-05-30T21:38:45.992ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
71832025-05-30T21:38:45.992ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
71842025-05-30T21:38:45.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
71852025-05-30T21:38:45.993ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71862025-05-30T21:38:45.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71872025-05-30T21:38:45.995ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
71882025-05-30T21:38:45.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71892025-05-30T21:38:45.998ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
71902025-05-30T21:38:45.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71912025-05-30T21:38:45.999ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71922025-05-30T21:38:45.999ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71932025-05-30T21:38:46.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
71942025-05-30T21:38:46.002ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
71952025-05-30T21:38:46.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71962025-05-30T21:38:46.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
71972025-05-30T21:38:46.003ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
71982025-05-30T21:38:46.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
71992025-05-30T21:38:46.003ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 60.661927ms iteration = 26 status = {"errors":[],"requests_created_ok":[]}
72002025-05-30T21:38:46.003ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
72012025-05-30T21:38:46.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
72022025-05-30T21:38:46.004ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
72032025-05-30T21:38:46.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
72042025-05-30T21:38:46.005ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
72052025-05-30T21:38:46.006ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 62.984304ms iteration = 26 status = 0
72062025-05-30T21:38:46.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
72072025-05-30T21:38:46.007ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72082025-05-30T21:38:46.009ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72092025-05-30T21:38:46.009ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72102025-05-30T21:38:46.009ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
72112025-05-30T21:38:46.009ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72122025-05-30T21:38:46.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
72132025-05-30T21:38:46.010ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
72142025-05-30T21:38:46.011ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72152025-05-30T21:38:46.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
72162025-05-30T21:38:46.013ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 69.881993ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
72172025-05-30T21:38:46.013ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
72182025-05-30T21:38:46.013ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72192025-05-30T21:38:46.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72202025-05-30T21:38:46.013ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
72212025-05-30T21:38:46.013ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72222025-05-30T21:38:46.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
72232025-05-30T21:38:46.014ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
72242025-05-30T21:38:46.014ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
72252025-05-30T21:38:46.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72262025-05-30T21:38:46.015ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
72272025-05-30T21:38:46.015ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 72.020977ms iteration = 26 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
72282025-05-30T21:38:46.015ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72292025-05-30T21:38:46.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72302025-05-30T21:38:46.017ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72312025-05-30T21:38:46.020ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
72322025-05-30T21:38:46.020ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72332025-05-30T21:38:46.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72342025-05-30T21:38:46.022ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
72352025-05-30T21:38:46.022ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 79.880068ms iteration = 76 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
72362025-05-30T21:38:46.037ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 94.990991ms iteration = 76 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
72372025-05-30T21:38:46.038ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
72382025-05-30T21:38:46.038ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
72392025-05-30T21:38:46.038ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
72402025-05-30T21:38:46.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
72412025-05-30T21:38:46.039ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
72422025-05-30T21:38:46.040ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 96.724931ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
72432025-05-30T21:38:46.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
72442025-05-30T21:38:46.041ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72452025-05-30T21:38:46.042ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 99.3069ms iteration = 26 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
72462025-05-30T21:38:46.043ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
72472025-05-30T21:38:46.043ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
72482025-05-30T21:38:46.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
72492025-05-30T21:38:46.076ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
72502025-05-30T21:38:46.076ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
72512025-05-30T21:38:46.077ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb96e612-314f-4658-8872-15e6202d404b", "content-length": "26", "date": "Fri, 30 May 2025 21:38:46 GMT"} })
72522025-05-30T21:38:46.077ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
72532025-05-30T21:38:46.079ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
72542025-05-30T21:38:46.079ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
72552025-05-30T21:38:46.079ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4c248ba-08ff-4069-a2eb-7695f398f7a0", "content-length": "26", "date": "Fri, 30 May 2025 21:38:46 GMT"} })
72562025-05-30T21:38:46.079ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
72572025-05-30T21:38:46.079ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
72582025-05-30T21:38:46.081ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
72592025-05-30T21:38:46.091ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72602025-05-30T21:38:46.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72612025-05-30T21:38:46.103ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
72622025-05-30T21:38:46.103ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
72632025-05-30T21:38:46.103ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 160.534454ms iteration = 26 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
72642025-05-30T21:38:46.110ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
72652025-05-30T21:38:46.110ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 167.305331ms iteration = 26 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
72662025-05-30T21:38:46.168ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 225.022175ms iteration = 28 status = {"physical_disks_added":0}
72672025-05-30T21:38:46.217ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 274.177289ms iteration = 26 status = {}
72682025-05-30T21:38:46.217ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
72692025-05-30T21:38:46.217ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
72702025-05-30T21:38:46.217ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a4e4577-f36c-4b8b-8a0d-8609ef6807db", "content-length": "26", "date": "Fri, 30 May 2025 21:38:46 GMT"} })
72712025-05-30T21:38:46.217ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
72722025-05-30T21:38:46.219ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
72732025-05-30T21:38:46.222ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72742025-05-30T21:38:46.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72752025-05-30T21:38:46.224ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72762025-05-30T21:38:46.226ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
72772025-05-30T21:38:46.226ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72782025-05-30T21:38:46.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72792025-05-30T21:38:46.227ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72802025-05-30T21:38:46.229ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
72812025-05-30T21:38:46.229ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72822025-05-30T21:38:46.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72832025-05-30T21:38:46.240ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
72842025-05-30T21:38:46.240ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
72852025-05-30T21:38:46.240ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "925e676d-54f3-44d8-bf1c-068a4d474f49", "content-length": "372", "date": "Fri, 30 May 2025 21:38:45 GMT"} })
72862025-05-30T21:38:46.240ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72872025-05-30T21:38:46.240ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
72882025-05-30T21:38:46.255ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
72892025-05-30T21:38:46.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
72902025-05-30T21:38:46.257ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 314.614074ms iteration = 26 status = null
72912025-05-30T21:38:46.288ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
72922025-05-30T21:38:46.288ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
72932025-05-30T21:38:46.288ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "502a1e51-7036-40ec-9237-94f55534ccbe", "date": "Fri, 30 May 2025 21:38:45 GMT"} })
72942025-05-30T21:38:46.288ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72952025-05-30T21:38:46.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
72962025-05-30T21:38:46.289ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72972025-05-30T21:38:46.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
72982025-05-30T21:38:46.292ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72992025-05-30T21:38:46.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73002025-05-30T21:38:46.296ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
73012025-05-30T21:38:46.296ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
73022025-05-30T21:38:46.296ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "dc465741-33c2-405a-9c36-6ca9e55e250d", "date": "Fri, 30 May 2025 21:38:45 GMT"} })
73032025-05-30T21:38:46.297ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 354.977087ms iteration = 26 status = {}
73042025-05-30T21:38:46.349ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73052025-05-30T21:38:46.349ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73062025-05-30T21:38:46.351ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
73072025-05-30T21:38:46.351ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73082025-05-30T21:38:46.351ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73092025-05-30T21:38:46.360ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
73102025-05-30T21:38:46.360ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "078c1390-e9ad-4e95-b5cc-0fded77e9f86", "content-length": "691", "date": "Fri, 30 May 2025 21:38:45 GMT"} })
73112025-05-30T21:38:46.360ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
73122025-05-30T21:38:46.360ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
73132025-05-30T21:38:46.360ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
73142025-05-30T21:38:46.362ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
73152025-05-30T21:38:46.362ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
73162025-05-30T21:38:46.362ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
73172025-05-30T21:38:46.362ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
73182025-05-30T21:38:46.362ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
73192025-05-30T21:38:46.362ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
73202025-05-30T21:38:46.362ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
73212025-05-30T21:38:46.362ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
73222025-05-30T21:38:46.362ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 419.642289ms iteration = 26 status = {}
73232025-05-30T21:38:46.362ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
73242025-05-30T21:38:46.362ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
73252025-05-30T21:38:46.363ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
73262025-05-30T21:38:46.363ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73272025-05-30T21:38:46.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73282025-05-30T21:38:46.365ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
73292025-05-30T21:38:46.365ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
73302025-05-30T21:38:46.477ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "68ee4788-cc18-473e-95ba-a366771af425", "date": "Fri, 30 May 2025 21:38:45 GMT"} })
73312025-05-30T21:38:46.477ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
73322025-05-30T21:38:46.477ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
73332025-05-30T21:38:46.481ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "0915fafc-ba40-49fc-b2c2-fd62fb2a1ddb", "content-length": "682", "date": "Fri, 30 May 2025 21:38:46 GMT"} })
73342025-05-30T21:38:46.481ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73352025-05-30T21:38:46.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73362025-05-30T21:38:46.486ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73372025-05-30T21:38:46.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73382025-05-30T21:38:46.489ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73392025-05-30T21:38:46.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73402025-05-30T21:38:46.492ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73412025-05-30T21:38:46.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73422025-05-30T21:38:46.499ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 556.662044ms iteration = 26 status = {}
73432025-05-30T21:38:48.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 364 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 0775f2b7-2779-448a-8319-dc0e6db98092 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
73442025-05-30T21:38:55.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 77 reason = Timeout
73452025-05-30T21:38:55.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
73462025-05-30T21:38:55.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 77 reason = Timeout
73472025-05-30T21:38:55.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73482025-05-30T21:38:55.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
73492025-05-30T21:38:55.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73502025-05-30T21:38:55.945ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
73512025-05-30T21:38:55.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73522025-05-30T21:38:55.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
73532025-05-30T21:38:55.947ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.996186ms iteration = 77 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
73542025-05-30T21:38:55.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
73552025-05-30T21:38:55.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73562025-05-30T21:38:55.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73572025-05-30T21:38:55.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73582025-05-30T21:38:55.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73592025-05-30T21:38:55.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73602025-05-30T21:38:55.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
73612025-05-30T21:38:55.955ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73622025-05-30T21:38:55.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73632025-05-30T21:38:55.965ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
73642025-05-30T21:38:55.965ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.943194ms iteration = 77 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
73652025-05-30T21:38:58.779ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 364 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = d79f8fe1-b1a3-470f-8668-6a7eae6c7d93 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
73662025-05-30T21:39:05.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 78 reason = Timeout
73672025-05-30T21:39:05.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73682025-05-30T21:39:05.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 78 reason = Timeout
73692025-05-30T21:39:05.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
73702025-05-30T21:39:05.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73712025-05-30T21:39:05.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
73722025-05-30T21:39:05.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73732025-05-30T21:39:05.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
73742025-05-30T21:39:05.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73752025-05-30T21:39:05.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73762025-05-30T21:39:05.953ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73772025-05-30T21:39:05.953ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
73782025-05-30T21:39:05.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73792025-05-30T21:39:05.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
73802025-05-30T21:39:05.955ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73812025-05-30T21:39:05.956ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 12.689205ms iteration = 78 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
73822025-05-30T21:39:05.958ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
73832025-05-30T21:39:05.958ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73842025-05-30T21:39:05.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73852025-05-30T21:39:05.970ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
73862025-05-30T21:39:05.970ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.223083ms iteration = 78 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
73872025-05-30T21:39:08.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 348 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = ea77cde3-4a10-42b4-aeda-329c2843ef0a response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
73882025-05-30T21:39:15.942ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 14 reason = Timeout
73892025-05-30T21:39:15.942ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
73902025-05-30T21:39:15.942ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
73912025-05-30T21:39:15.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
73922025-05-30T21:39:15.942ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 79 reason = Timeout
73932025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73942025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 27 reason = Timeout
73952025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 27 reason = Timeout
73962025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
73972025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
73982025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
73992025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
74002025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 27 reason = Timeout
74012025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
74022025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 27 reason = Timeout
74032025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
74042025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 27 reason = Timeout
74052025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
74062025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 27 reason = Timeout
74072025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
74082025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 27 reason = Timeout
74092025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
74102025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 27 reason = Timeout
74112025-05-30T21:39:15.943ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
74122025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
74132025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 27 reason = Timeout
74142025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74152025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 79 reason = Timeout
74162025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74172025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74182025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 29 reason = Timeout
74192025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
74202025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 27 reason = Timeout
74212025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 27 reason = Timeout
74222025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
74232025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 14 reason = Timeout
74242025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
74252025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
74262025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 14 reason = Timeout
74272025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
74282025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 14 reason = Timeout
74292025-05-30T21:39:15.943ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-30 21:29:15.943257367 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
74302025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
74312025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 14 reason = Timeout
74322025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
74332025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 14 reason = Timeout
74342025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
74352025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 14 reason = Timeout
74362025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
74372025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 16 reason = Timeout
74382025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 14 reason = Timeout
74392025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
74402025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
74412025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 27 reason = Timeout
74422025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 15 reason = Timeout
74432025-05-30T21:39:15.943ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e
74442025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 14 reason = Timeout
74452025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.03\u{b5}s iteration = 15 status = {"enabled":false,"target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e"}
74462025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 27 reason = Timeout
74472025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 14 reason = Timeout
74482025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
74492025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
74502025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 16 reason = Timeout
74512025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 27 reason = Timeout
74522025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
74532025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 14 reason = Timeout
74542025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 27 reason = Timeout
74552025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 27 reason = Timeout
74562025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
74572025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
74582025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 14 reason = Timeout
74592025-05-30T21:39:15.943ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
74602025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
74612025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
74622025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
74632025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74642025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
74652025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
74662025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
74672025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
74682025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
74692025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
74702025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
74712025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
74722025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
74732025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74742025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
74752025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
74762025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
74772025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
74782025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
74792025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
74802025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
74812025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
74822025-05-30T21:39:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
74832025-05-30T21:39:15.943ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2a161087-f464-41f8-b020-0f9e83fd5351.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1890, tv_nsec: 878997164 } })
74842025-05-30T21:39:15.944ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0413f6b4-86a1-42a1-919a-e8f42ff9f638.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("acb1452d-0a02-4b36-a462-c8ce2f96a488.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:2::1))) }], valid_until: Instant { tv_sec: 1890, tv_nsec: 879007885 } })
74852025-05-30T21:39:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
74862025-05-30T21:39:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
74872025-05-30T21:39:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
74882025-05-30T21:39:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
74892025-05-30T21:39:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
74902025-05-30T21:39:15.944ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
74912025-05-30T21:39:15.944ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.052682ms iteration = 14 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
74922025-05-30T21:39:15.944ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } dns_group = internal
74932025-05-30T21:39:15.944ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.044242ms iteration = 14 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353"]}
74942025-05-30T21:39:15.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74952025-05-30T21:39:15.946ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1890, tv_nsec: 881616164 } })
74962025-05-30T21:39:15.946ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
74972025-05-30T21:39:15.947ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74982025-05-30T21:39:15.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74992025-05-30T21:39:15.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75002025-05-30T21:39:15.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75012025-05-30T21:39:15.948ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.268891ms iteration = 27 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
75022025-05-30T21:39:15.949ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
75032025-05-30T21:39:15.949ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
75042025-05-30T21:39:15.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
75052025-05-30T21:39:15.950ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 7.371684ms iteration = 27 status = {"errors":[],"garbage_collect_requested":[]}
75062025-05-30T21:39:15.951ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75072025-05-30T21:39:15.952ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
75082025-05-30T21:39:15.952ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
75092025-05-30T21:39:15.952ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
75102025-05-30T21:39:15.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
75112025-05-30T21:39:15.953ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
75122025-05-30T21:39:15.953ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 10.192137ms iteration = 14 status = {"errors":[],"found_port_ok":[]}
75132025-05-30T21:39:15.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
75142025-05-30T21:39:15.954ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
75152025-05-30T21:39:15.956ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1890, tv_nsec: 891874372 } })
75162025-05-30T21:39:15.957ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75172025-05-30T21:39:15.957ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
75182025-05-30T21:39:15.959ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:202 num_receivers = 0
75192025-05-30T21:39:15.959ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-30 21:29:15.943257367 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
75202025-05-30T21:39:15.959ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 15.963573ms iteration = 14 status = {"by_rx":{},"error":null}
75212025-05-30T21:39:15.959ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 16.001974ms iteration = 14 status = {"expiration":"2025-05-30T21:29:15.943257367Z","pruned":[]}
75222025-05-30T21:39:15.960ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
75232025-05-30T21:39:15.960ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75242025-05-30T21:39:15.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75252025-05-30T21:39:15.961ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
75262025-05-30T21:39:15.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
75272025-05-30T21:39:15.962ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75282025-05-30T21:39:15.963ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75292025-05-30T21:39:15.964ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75302025-05-30T21:39:15.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75312025-05-30T21:39:15.966ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1890, tv_nsec: 901017597 } })
75322025-05-30T21:39:15.966ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
75332025-05-30T21:39:15.967ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
75342025-05-30T21:39:15.967ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
75352025-05-30T21:39:15.967ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
75362025-05-30T21:39:15.967ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75372025-05-30T21:39:15.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
75382025-05-30T21:39:15.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75392025-05-30T21:39:15.970ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75402025-05-30T21:39:15.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75412025-05-30T21:39:15.971ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
75422025-05-30T21:39:15.972ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75432025-05-30T21:39:15.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
75442025-05-30T21:39:15.972ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75452025-05-30T21:39:15.975ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
75462025-05-30T21:39:15.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
75472025-05-30T21:39:15.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75482025-05-30T21:39:15.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75492025-05-30T21:39:15.981ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
75502025-05-30T21:39:15.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
75512025-05-30T21:39:15.982ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75522025-05-30T21:39:15.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75532025-05-30T21:39:15.988ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75542025-05-30T21:39:15.990ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
75552025-05-30T21:39:15.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
75562025-05-30T21:39:15.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
75572025-05-30T21:39:15.991ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75582025-05-30T21:39:15.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75592025-05-30T21:39:15.994ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 51.724144ms iteration = 14 status = {"dispatched":[],"errors":[],"glob_version":"144.0.0","globs_reprocessed":{},"no_receivers":[]}
75602025-05-30T21:39:15.995ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75612025-05-30T21:39:15.995ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75622025-05-30T21:39:15.998ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
75632025-05-30T21:39:15.998ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75642025-05-30T21:39:15.998ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 54.665638ms iteration = 27 status = {"errors":[],"finish_invoked_ok":[]}
75652025-05-30T21:39:15.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75662025-05-30T21:39:15.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
75672025-05-30T21:39:15.999ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
75682025-05-30T21:39:15.999ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 56.406178ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
75692025-05-30T21:39:15.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
75702025-05-30T21:39:15.999ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
75712025-05-30T21:39:16.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
75722025-05-30T21:39:16.000ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
75732025-05-30T21:39:16.000ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75742025-05-30T21:39:16.000ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 57.595501ms iteration = 27 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
75752025-05-30T21:39:16.000ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
75762025-05-30T21:39:16.000ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 57.928686ms iteration = 79 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
75772025-05-30T21:39:16.001ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75782025-05-30T21:39:16.003ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
75792025-05-30T21:39:16.003ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75802025-05-30T21:39:16.003ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75812025-05-30T21:39:16.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75822025-05-30T21:39:16.005ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
75832025-05-30T21:39:16.005ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75842025-05-30T21:39:16.005ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
75852025-05-30T21:39:16.005ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
75862025-05-30T21:39:16.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75872025-05-30T21:39:16.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
75882025-05-30T21:39:16.005ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
75892025-05-30T21:39:16.005ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
75902025-05-30T21:39:16.005ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
75912025-05-30T21:39:16.005ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
75922025-05-30T21:39:16.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
75932025-05-30T21:39:16.008ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75942025-05-30T21:39:16.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75952025-05-30T21:39:16.008ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75962025-05-30T21:39:16.009ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75972025-05-30T21:39:16.009ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75982025-05-30T21:39:16.010ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
75992025-05-30T21:39:16.010ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
76002025-05-30T21:39:16.010ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
76012025-05-30T21:39:16.010ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
76022025-05-30T21:39:16.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
76032025-05-30T21:39:16.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
76042025-05-30T21:39:16.012ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
76052025-05-30T21:39:16.012ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
76062025-05-30T21:39:16.012ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 69.58356ms iteration = 14 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
76072025-05-30T21:39:16.012ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 69.827092ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
76082025-05-30T21:39:16.013ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
76092025-05-30T21:39:16.013ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
76102025-05-30T21:39:16.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
76112025-05-30T21:39:16.013ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
76122025-05-30T21:39:16.013ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
76132025-05-30T21:39:16.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
76142025-05-30T21:39:16.014ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
76152025-05-30T21:39:16.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
76162025-05-30T21:39:16.014ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
76172025-05-30T21:39:16.014ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
76182025-05-30T21:39:16.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
76192025-05-30T21:39:16.020ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 77.276858ms iteration = 27 status = {"errors":[],"requests_created_ok":[]}
76202025-05-30T21:39:16.026ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
76212025-05-30T21:39:16.026ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
76222025-05-30T21:39:16.026ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
76232025-05-30T21:39:16.026ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
76242025-05-30T21:39:16.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
76252025-05-30T21:39:16.030ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76262025-05-30T21:39:16.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76272025-05-30T21:39:16.035ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
76282025-05-30T21:39:16.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
76292025-05-30T21:39:16.037ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
76302025-05-30T21:39:16.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
76312025-05-30T21:39:16.054ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
76322025-05-30T21:39:16.054ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
76332025-05-30T21:39:16.054ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 110.927564ms iteration = 14 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
76342025-05-30T21:39:16.079ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
76352025-05-30T21:39:16.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
76362025-05-30T21:39:16.090ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
76372025-05-30T21:39:16.090ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
76382025-05-30T21:39:16.090ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "229dc2c0-6dcd-45e6-bdbd-a34231849242", "content-length": "26", "date": "Fri, 30 May 2025 21:39:16 GMT"} })
76392025-05-30T21:39:16.090ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
76402025-05-30T21:39:16.093ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
76412025-05-30T21:39:16.099ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
76422025-05-30T21:39:16.099ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
76432025-05-30T21:39:16.099ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d72b3e8a-0856-4a0c-9199-a4b28459e2f8", "content-length": "26", "date": "Fri, 30 May 2025 21:39:16 GMT"} })
76442025-05-30T21:39:16.099ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
76452025-05-30T21:39:16.102ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
76462025-05-30T21:39:16.122ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
76472025-05-30T21:39:16.122ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
76482025-05-30T21:39:16.123ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b06f1508-51de-4c03-98ce-3d93fb7dba41", "content-length": "26", "date": "Fri, 30 May 2025 21:39:16 GMT"} })
76492025-05-30T21:39:16.123ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
76502025-05-30T21:39:16.125ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
76512025-05-30T21:39:16.186ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
76522025-05-30T21:39:16.187ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
76532025-05-30T21:39:16.187ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "a67a3ae5-83e8-4baf-a480-58974c447429", "content-length": "372", "date": "Fri, 30 May 2025 21:39:15 GMT"} })
76542025-05-30T21:39:16.187ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76552025-05-30T21:39:16.187ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76562025-05-30T21:39:16.189ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 246.610882ms iteration = 79 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
76572025-05-30T21:39:16.193ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76582025-05-30T21:39:16.194ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76592025-05-30T21:39:16.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76602025-05-30T21:39:16.196ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
76612025-05-30T21:39:16.196ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
76622025-05-30T21:39:16.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
76632025-05-30T21:39:16.197ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
76642025-05-30T21:39:16.197ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 254.186208ms iteration = 27 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
76652025-05-30T21:39:16.206ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
76662025-05-30T21:39:16.222ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
76672025-05-30T21:39:16.232ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
76682025-05-30T21:39:16.232ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
76692025-05-30T21:39:16.232ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 289.062059ms iteration = 27 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
76702025-05-30T21:39:16.242ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76712025-05-30T21:39:16.246ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
76722025-05-30T21:39:16.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
76732025-05-30T21:39:16.247ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
76742025-05-30T21:39:16.247ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76752025-05-30T21:39:16.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76762025-05-30T21:39:16.248ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 305.623819ms iteration = 27 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
76772025-05-30T21:39:16.249ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 305.936174ms iteration = 27 status = null
76782025-05-30T21:39:16.249ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 305.826972ms iteration = 27 status = 0
76792025-05-30T21:39:16.251ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76802025-05-30T21:39:16.251ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
76812025-05-30T21:39:16.251ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
76822025-05-30T21:39:16.251ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
76832025-05-30T21:39:16.252ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
76842025-05-30T21:39:16.253ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76852025-05-30T21:39:16.254ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-30T21:26:13.986396Z, time_modified: 2025-05-30T21:26:13.986396Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }) }
76862025-05-30T21:39:16.254ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 310.843909ms iteration = 14 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]}},"default_endpoint":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
76872025-05-30T21:39:16.256ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
76882025-05-30T21:39:16.256ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
76892025-05-30T21:39:16.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
76902025-05-30T21:39:16.257ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
76912025-05-30T21:39:16.257ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
76922025-05-30T21:39:16.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
76932025-05-30T21:39:16.260ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
76942025-05-30T21:39:16.260ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
76952025-05-30T21:39:16.260ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-30 21:26:17.226115 UTC dns_group = internal generation = 1
76962025-05-30T21:39:16.260ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 318.857681ms iteration = 14 status = {"generation":1}
76972025-05-30T21:39:16.262ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
76982025-05-30T21:39:16.262ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
76992025-05-30T21:39:16.262ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "9fa08841-3dff-4173-b10e-d3791cad6ecc", "date": "Fri, 30 May 2025 21:39:15 GMT"} })
77002025-05-30T21:39:16.262ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77012025-05-30T21:39:16.262ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77022025-05-30T21:39:16.264ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77032025-05-30T21:39:16.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77042025-05-30T21:39:16.268ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77052025-05-30T21:39:16.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77062025-05-30T21:39:16.270ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 327.028725ms iteration = 27 status = {}
77072025-05-30T21:39:16.272ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
77082025-05-30T21:39:16.272ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
77092025-05-30T21:39:16.272ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "712b868d-10a7-47f5-a871-2a9dfbd8cb56", "date": "Fri, 30 May 2025 21:39:15 GMT"} })
77102025-05-30T21:39:16.274ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 330.983991ms iteration = 27 status = {}
77112025-05-30T21:39:16.298ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
77122025-05-30T21:39:16.298ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 0 version = 2
77132025-05-30T21:39:16.298ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
77142025-05-30T21:39:16.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
77152025-05-30T21:39:16.306ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77162025-05-30T21:39:16.309ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
77172025-05-30T21:39:16.309ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
77182025-05-30T21:39:16.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
77192025-05-30T21:39:16.311ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 1 version = 2
77202025-05-30T21:39:16.311ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
77212025-05-30T21:39:16.311ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-30 21:26:18.957109 UTC dns_group = external generation = 2
77222025-05-30T21:39:16.311ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 368.107377ms iteration = 14 status = {"generation":2}
77232025-05-30T21:39:16.329ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 386.67351ms iteration = 29 status = {"physical_disks_added":0}
77242025-05-30T21:39:16.329ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
77252025-05-30T21:39:16.330ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
77262025-05-30T21:39:16.363ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77272025-05-30T21:39:16.367ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
77282025-05-30T21:39:16.367ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 423.779577ms iteration = 14 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
77292025-05-30T21:39:16.367ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
77302025-05-30T21:39:16.367ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
77312025-05-30T21:39:16.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
77322025-05-30T21:39:16.399ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77332025-05-30T21:39:16.401ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
77342025-05-30T21:39:16.401ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
77352025-05-30T21:39:16.401ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
77362025-05-30T21:39:16.415ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
77372025-05-30T21:39:16.431ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
77382025-05-30T21:39:16.432ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "e071fe0f-b7cf-44e6-9d5b-bee1e1e74b82", "date": "Fri, 30 May 2025 21:39:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
77392025-05-30T21:39:16.432ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
77402025-05-30T21:39:16.434ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "ec06cf75-7aca-4032-8f70-88ba71cbcdaf", "date": "Fri, 30 May 2025 21:39:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
77412025-05-30T21:39:16.434ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
77422025-05-30T21:39:16.437ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
77432025-05-30T21:39:16.437ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 493.825901ms iteration = 16 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
77442025-05-30T21:39:16.475ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77452025-05-30T21:39:16.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77462025-05-30T21:39:16.477ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
77472025-05-30T21:39:16.477ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77482025-05-30T21:39:16.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77492025-05-30T21:39:16.490ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
77502025-05-30T21:39:16.491ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "c574e8e6-bbde-47f9-abb7-ae1cd12f916e", "date": "Fri, 30 May 2025 21:39:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
77512025-05-30T21:39:16.491ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
77522025-05-30T21:39:16.493ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "0e766ec5-e1ba-47f2-bda5-8bec6768ccee", "date": "Fri, 30 May 2025 21:39:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
77532025-05-30T21:39:16.493ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
77542025-05-30T21:39:16.494ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "86c87bab-cd72-4b96-958f-f5fe871f3cc6", "date": "Fri, 30 May 2025 21:39:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
77552025-05-30T21:39:16.494ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
77562025-05-30T21:39:16.496ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
77572025-05-30T21:39:16.496ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 553.472207ms iteration = 16 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
77582025-05-30T21:39:16.510ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
77592025-05-30T21:39:16.510ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "6586313c-0878-4976-a969-f71971432adb", "content-length": "691", "date": "Fri, 30 May 2025 21:39:15 GMT"} })
77602025-05-30T21:39:16.510ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
77612025-05-30T21:39:16.510ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
77622025-05-30T21:39:16.510ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
77632025-05-30T21:39:16.511ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 568.524669ms iteration = 27 status = {}
77642025-05-30T21:39:16.522ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
77652025-05-30T21:39:16.522ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
77662025-05-30T21:39:16.523ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
77672025-05-30T21:39:16.523ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
77682025-05-30T21:39:16.523ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77692025-05-30T21:39:16.523ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77702025-05-30T21:39:16.523ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77712025-05-30T21:39:16.523ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
77722025-05-30T21:39:16.523ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77732025-05-30T21:39:16.523ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
77742025-05-30T21:39:16.523ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
77752025-05-30T21:39:16.523ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77762025-05-30T21:39:16.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77772025-05-30T21:39:16.525ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
77782025-05-30T21:39:16.525ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
77792025-05-30T21:39:16.623ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "8271bd40-7b5c-4c80-893d-bae02fc23637", "date": "Fri, 30 May 2025 21:39:15 GMT"} })
77802025-05-30T21:39:16.623ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
77812025-05-30T21:39:16.623ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
77822025-05-30T21:39:16.627ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "693923f8-a8f7-4610-9641-9be08f4e64df", "content-length": "682", "date": "Fri, 30 May 2025 21:39:16 GMT"} })
77832025-05-30T21:39:16.627ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77842025-05-30T21:39:16.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77852025-05-30T21:39:16.631ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77862025-05-30T21:39:16.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77872025-05-30T21:39:16.634ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77882025-05-30T21:39:16.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77892025-05-30T21:39:16.636ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77902025-05-30T21:39:16.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77912025-05-30T21:39:16.643ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 700.093981ms iteration = 27 status = {}
77922025-05-30T21:39:18.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 358 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 1dde5a2a-e656-486a-9121-f5a5672442c7 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
77932025-05-30T21:39:25.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 80 reason = Timeout
77942025-05-30T21:39:25.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77952025-05-30T21:39:25.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 80 reason = Timeout
77962025-05-30T21:39:25.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77972025-05-30T21:39:25.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77982025-05-30T21:39:25.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77992025-05-30T21:39:25.978ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78002025-05-30T21:39:25.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
78012025-05-30T21:39:25.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78022025-05-30T21:39:25.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78032025-05-30T21:39:25.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
78042025-05-30T21:39:25.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
78052025-05-30T21:39:25.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78062025-05-30T21:39:25.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78072025-05-30T21:39:25.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78082025-05-30T21:39:25.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
78092025-05-30T21:39:25.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78102025-05-30T21:39:25.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78112025-05-30T21:39:25.998ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
78122025-05-30T21:39:25.998ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 54.88539ms iteration = 80 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
78132025-05-30T21:39:26.015ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 72.389781ms iteration = 80 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
78142025-05-30T21:39:28.779ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 337 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 0c65dabd-51fa-4673-be17-cf69659f9028 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
78152025-05-30T21:39:35.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 81 reason = Timeout
78162025-05-30T21:39:35.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 81 reason = Timeout
78172025-05-30T21:39:35.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
78182025-05-30T21:39:35.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78192025-05-30T21:39:35.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78202025-05-30T21:39:35.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
78212025-05-30T21:39:35.945ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78222025-05-30T21:39:35.945ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
78232025-05-30T21:39:35.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
78242025-05-30T21:39:35.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
78252025-05-30T21:39:35.948ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78262025-05-30T21:39:35.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78272025-05-30T21:39:35.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78282025-05-30T21:39:35.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78292025-05-30T21:39:35.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78302025-05-30T21:39:35.954ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
78312025-05-30T21:39:35.954ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78322025-05-30T21:39:35.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78332025-05-30T21:39:35.966ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
78342025-05-30T21:39:35.966ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.222256ms iteration = 81 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
78352025-05-30T21:39:35.978ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 35.859592ms iteration = 81 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
78362025-05-30T21:39:38.779ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 293 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = bf4318f0-f770-4644-8829-76422751c0d2 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
78372025-05-30T21:39:43.769ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:48915
78382025-05-30T21:39:43.769ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48915 req_id = 9322c3d2-388a-41d1-92dc-1c747ee881e2 roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
78392025-05-30T21:39:43.769ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48915 req_id = 9322c3d2-388a-41d1-92dc-1c747ee881e2 resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
78402025-05-30T21:39:43.772ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3422 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48915 req_id = 9322c3d2-388a-41d1-92dc-1c747ee881e2 response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
78412025-05-30T21:39:43.773ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48915 req_id = 7a79eafe-763f-4500-bec4-a8b315add7d7 roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
78422025-05-30T21:39:43.773ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48915 req_id = 7a79eafe-763f-4500-bec4-a8b315add7d7 resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
78432025-05-30T21:39:43.775ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2413 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48915 req_id = 7a79eafe-763f-4500-bec4-a8b315add7d7 response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
78442025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 82 reason = Timeout
78452025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 28 reason = Timeout
78462025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
78472025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
78482025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
78492025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 28 reason = Timeout
78502025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
78512025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 28 reason = Timeout
78522025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
78532025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 28 reason = Timeout
78542025-05-30T21:39:45.943ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
78552025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 28 reason = Timeout
78562025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78572025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
78582025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 28 reason = Timeout
78592025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 30 reason = Timeout
78602025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
78612025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
78622025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 28 reason = Timeout
78632025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 82 reason = Timeout
78642025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78652025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78662025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 28 reason = Timeout
78672025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 28 reason = Timeout
78682025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
78692025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 28 reason = Timeout
78702025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
78712025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
78722025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 28 reason = Timeout
78732025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
78742025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 28 reason = Timeout
78752025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
78762025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 28 reason = Timeout
78772025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
78782025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 28 reason = Timeout
78792025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 28 reason = Timeout
78802025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
78812025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
78822025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 28 reason = Timeout
78832025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
78842025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
78852025-05-30T21:39:45.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
78862025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78872025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
78882025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
78892025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
78902025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
78912025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78922025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
78932025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
78942025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
78952025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78962025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
78972025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
78982025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
78992025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
79002025-05-30T21:39:45.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
79012025-05-30T21:39:45.946ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.732342ms iteration = 28 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
79022025-05-30T21:39:45.946ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
79032025-05-30T21:39:45.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
79042025-05-30T21:39:45.948ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79052025-05-30T21:39:45.948ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
79062025-05-30T21:39:45.948ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
79072025-05-30T21:39:45.948ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
79082025-05-30T21:39:45.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
79092025-05-30T21:39:45.950ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1920, tv_nsec: 884831724 } })
79102025-05-30T21:39:45.950ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
79112025-05-30T21:39:45.950ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
79122025-05-30T21:39:45.950ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
79132025-05-30T21:39:45.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
79142025-05-30T21:39:45.954ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1920, tv_nsec: 889703020 } })
79152025-05-30T21:39:45.955ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
79162025-05-30T21:39:45.959ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
79172025-05-30T21:39:45.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
79182025-05-30T21:39:45.990ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
79192025-05-30T21:39:45.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
79202025-05-30T21:39:45.991ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79212025-05-30T21:39:45.993ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
79222025-05-30T21:39:45.993ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
79232025-05-30T21:39:45.993ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "572c4b10-3ebe-4ae8-a003-e425ff54aa38", "content-length": "26", "date": "Fri, 30 May 2025 21:39:45 GMT"} })
79242025-05-30T21:39:45.993ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
79252025-05-30T21:39:45.993ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79262025-05-30T21:39:45.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
79272025-05-30T21:39:45.994ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
79282025-05-30T21:39:45.994ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
79292025-05-30T21:39:45.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
79302025-05-30T21:39:45.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
79312025-05-30T21:39:45.995ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 51.31556ms iteration = 28 status = {"errors":[],"garbage_collect_requested":[]}
79322025-05-30T21:39:45.996ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
79332025-05-30T21:39:45.997ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 53.755067ms iteration = 28 status = {"errors":[],"requests_created_ok":[]}
79342025-05-30T21:39:45.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
79352025-05-30T21:39:45.998ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79362025-05-30T21:39:45.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79372025-05-30T21:39:45.999ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79382025-05-30T21:39:45.999ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
79392025-05-30T21:39:45.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
79402025-05-30T21:39:45.999ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
79412025-05-30T21:39:45.999ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
79422025-05-30T21:39:45.999ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
79432025-05-30T21:39:46.000ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79442025-05-30T21:39:46.000ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79452025-05-30T21:39:46.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
79462025-05-30T21:39:46.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
79472025-05-30T21:39:46.001ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
79482025-05-30T21:39:46.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
79492025-05-30T21:39:46.002ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
79502025-05-30T21:39:46.002ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
79512025-05-30T21:39:46.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
79522025-05-30T21:39:46.002ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
79532025-05-30T21:39:46.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
79542025-05-30T21:39:46.002ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0547b70a-ee75-4f24-8b5f-1c8392c4609b", "content-length": "26", "date": "Fri, 30 May 2025 21:39:45 GMT"} })
79552025-05-30T21:39:46.002ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
79562025-05-30T21:39:46.003ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79572025-05-30T21:39:46.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79582025-05-30T21:39:46.004ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
79592025-05-30T21:39:46.004ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
79602025-05-30T21:39:46.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
79612025-05-30T21:39:46.005ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
79622025-05-30T21:39:46.005ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.034722ms iteration = 82 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
79632025-05-30T21:39:46.005ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79642025-05-30T21:39:46.005ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 62.140514ms iteration = 28 status = {"errors":[],"finish_invoked_ok":[]}
79652025-05-30T21:39:46.006ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79662025-05-30T21:39:46.006ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1920, tv_nsec: 940819948 } })
79672025-05-30T21:39:46.006ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
79682025-05-30T21:39:46.008ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
79692025-05-30T21:39:46.008ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
79702025-05-30T21:39:46.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
79712025-05-30T21:39:46.008ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79722025-05-30T21:39:46.008ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
79732025-05-30T21:39:46.008ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
79742025-05-30T21:39:46.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
79752025-05-30T21:39:46.011ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
79762025-05-30T21:39:46.011ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79772025-05-30T21:39:46.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79782025-05-30T21:39:46.013ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79792025-05-30T21:39:46.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79802025-05-30T21:39:46.041ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
79812025-05-30T21:39:46.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
79822025-05-30T21:39:46.043ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79832025-05-30T21:39:46.044ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 101.424655ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
79842025-05-30T21:39:46.044ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 101.535067ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
79852025-05-30T21:39:46.046ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
79862025-05-30T21:39:46.046ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
79872025-05-30T21:39:46.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
79882025-05-30T21:39:46.056ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79892025-05-30T21:39:46.057ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79902025-05-30T21:39:46.058ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
79912025-05-30T21:39:46.058ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
79922025-05-30T21:39:46.058ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 115.439456ms iteration = 28 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
79932025-05-30T21:39:46.060ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79942025-05-30T21:39:46.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79952025-05-30T21:39:46.061ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
79962025-05-30T21:39:46.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
79972025-05-30T21:39:46.062ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79982025-05-30T21:39:46.062ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
79992025-05-30T21:39:46.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
80002025-05-30T21:39:46.063ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 119.580213ms iteration = 28 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
80012025-05-30T21:39:46.065ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80022025-05-30T21:39:46.067ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
80032025-05-30T21:39:46.067ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
80042025-05-30T21:39:46.067ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
80052025-05-30T21:39:46.069ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 125.623743ms iteration = 28 status = 0
80062025-05-30T21:39:46.069ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
80072025-05-30T21:39:46.069ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 126.033037ms iteration = 28 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
80082025-05-30T21:39:46.070ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
80092025-05-30T21:39:46.070ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
80102025-05-30T21:39:46.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
80112025-05-30T21:39:46.072ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 128.85264ms iteration = 28 status = {}
80122025-05-30T21:39:46.073ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
80132025-05-30T21:39:46.073ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
80142025-05-30T21:39:46.073ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "211c26ba-acda-43de-896a-ac4c3dfebbbf", "content-length": "26", "date": "Fri, 30 May 2025 21:39:46 GMT"} })
80152025-05-30T21:39:46.073ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
80162025-05-30T21:39:46.075ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
80172025-05-30T21:39:46.076ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
80182025-05-30T21:39:46.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
80192025-05-30T21:39:46.081ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 137.970085ms iteration = 28 status = null
80202025-05-30T21:39:46.089ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
80212025-05-30T21:39:46.089ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 145.907046ms iteration = 82 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
80222025-05-30T21:39:46.101ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
80232025-05-30T21:39:46.101ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 157.927764ms iteration = 28 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
80242025-05-30T21:39:46.150ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 206.529402ms iteration = 30 status = {"physical_disks_added":0}
80252025-05-30T21:39:46.151ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
80262025-05-30T21:39:46.151ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
80272025-05-30T21:39:46.151ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "03f5ee75-414a-4dd0-bbc3-121e292f5b7f", "content-length": "372", "date": "Fri, 30 May 2025 21:39:45 GMT"} })
80282025-05-30T21:39:46.151ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80292025-05-30T21:39:46.152ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80302025-05-30T21:39:46.173ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
80312025-05-30T21:39:46.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
80322025-05-30T21:39:46.175ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80332025-05-30T21:39:46.177ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
80342025-05-30T21:39:46.177ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
80352025-05-30T21:39:46.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
80362025-05-30T21:39:46.179ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80372025-05-30T21:39:46.181ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
80382025-05-30T21:39:46.181ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
80392025-05-30T21:39:46.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
80402025-05-30T21:39:46.201ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
80412025-05-30T21:39:46.201ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
80422025-05-30T21:39:46.201ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "f1fccf5d-689b-4eae-bac9-82715f52e5ab", "date": "Fri, 30 May 2025 21:39:45 GMT"} })
80432025-05-30T21:39:46.201ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80442025-05-30T21:39:46.202ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80452025-05-30T21:39:46.204ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80462025-05-30T21:39:46.204ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80472025-05-30T21:39:46.206ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80482025-05-30T21:39:46.206ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80492025-05-30T21:39:46.207ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80502025-05-30T21:39:46.207ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80512025-05-30T21:39:46.208ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
80522025-05-30T21:39:46.208ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80532025-05-30T21:39:46.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80542025-05-30T21:39:46.211ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
80552025-05-30T21:39:46.211ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
80562025-05-30T21:39:46.211ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "edcd2c5a-0451-4491-b296-5cd7b8420376", "date": "Fri, 30 May 2025 21:39:45 GMT"} })
80572025-05-30T21:39:46.212ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 269.432345ms iteration = 28 status = {}
80582025-05-30T21:39:46.219ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
80592025-05-30T21:39:46.219ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
80602025-05-30T21:39:46.219ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
80612025-05-30T21:39:46.219ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
80622025-05-30T21:39:46.219ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
80632025-05-30T21:39:46.219ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
80642025-05-30T21:39:46.219ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
80652025-05-30T21:39:46.219ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
80662025-05-30T21:39:46.219ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
80672025-05-30T21:39:46.219ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
80682025-05-30T21:39:46.219ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
80692025-05-30T21:39:46.219ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80702025-05-30T21:39:46.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80712025-05-30T21:39:46.222ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
80722025-05-30T21:39:46.222ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
80732025-05-30T21:39:46.273ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
80742025-05-30T21:39:46.323ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "50237f73-6b7a-475f-ac14-c3954d806384", "date": "Fri, 30 May 2025 21:39:45 GMT"} })
80752025-05-30T21:39:46.323ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
80762025-05-30T21:39:46.323ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
80772025-05-30T21:39:46.323ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "3add32ff-4b44-4f4a-a0a9-aa242b1d9220", "content-length": "682", "date": "Fri, 30 May 2025 21:39:45 GMT"} })
80782025-05-30T21:39:46.323ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80792025-05-30T21:39:46.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80802025-05-30T21:39:46.324ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "78ebf219-988f-4752-96bb-181b3c2838f1", "content-length": "691", "date": "Fri, 30 May 2025 21:39:46 GMT"} })
80812025-05-30T21:39:46.324ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
80822025-05-30T21:39:46.324ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
80832025-05-30T21:39:46.324ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
80842025-05-30T21:39:46.326ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 382.661825ms iteration = 28 status = {}
80852025-05-30T21:39:46.327ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80862025-05-30T21:39:46.328ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80872025-05-30T21:39:46.331ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80882025-05-30T21:39:46.331ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80892025-05-30T21:39:46.333ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80902025-05-30T21:39:46.333ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80912025-05-30T21:39:46.341ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 397.616837ms iteration = 28 status = {}
80922025-05-30T21:39:46.740ZDEBGnexus (producer-registration-task): registering as oximeter producer with Nexus registration_count = 5
80932025-05-30T21:39:46.740ZDEBGnexus (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221
80942025-05-30T21:39:46.769ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
80952025-05-30T21:39:46.769ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:49932
80962025-05-30T21:39:46.770ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:49932 req_id = 5d425293-d411-4e1f-ab1b-b38945ea1c56 roles = RoleSet { roles: {} } uri = /metrics/producers
80972025-05-30T21:39:46.770ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:49932 req_id = 5d425293-d411-4e1f-ab1b-b38945ea1c56 resource = Database result = Ok(()) uri = /metrics/producers
80982025-05-30T21:39:46.807ZINFOnexus: assigned collector to new producer collector_id = bc707af4-d9a3-402f-b2b6-642b29342fa8 file = nexus/src/app/oximeter.rs:76 producer_id = bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
80992025-05-30T21:39:46.807ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 37847 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:49932 req_id = 5d425293-d411-4e1f-ab1b-b38945ea1c56 response_code = 201 uri = /metrics/producers
81002025-05-30T21:39:46.807ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "5d425293-d411-4e1f-ab1b-b38945ea1c56", "content-length": "41", "date": "Fri, 30 May 2025 21:39:46 GMT"} })
81012025-05-30T21:39:46.809ZDEBGnexus (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
81022025-05-30T21:39:48.779ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 325 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 155feaa3-8a16-45ab-9852-d0c14d3be6a7 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
81032025-05-30T21:39:55.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 83 reason = Timeout
81042025-05-30T21:39:55.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81052025-05-30T21:39:55.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 83 reason = Timeout
81062025-05-30T21:39:55.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81072025-05-30T21:39:55.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81082025-05-30T21:39:55.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81092025-05-30T21:39:55.946ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81102025-05-30T21:39:55.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81112025-05-30T21:39:55.976ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81122025-05-30T21:39:55.977ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.876379ms iteration = 83 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
81132025-05-30T21:39:55.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
81142025-05-30T21:39:55.979ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81152025-05-30T21:39:55.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81162025-05-30T21:39:55.981ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81172025-05-30T21:39:55.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81182025-05-30T21:39:56.013ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81192025-05-30T21:39:56.016ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
81202025-05-30T21:39:56.016ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81212025-05-30T21:39:56.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81222025-05-30T21:39:56.029ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
81232025-05-30T21:39:56.029ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 85.997477ms iteration = 83 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
81242025-05-30T21:39:58.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 319 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 98c15db3-0f13-4475-8679-a42c15196357 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
81252025-05-30T21:40:05.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 84 reason = Timeout
81262025-05-30T21:40:05.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81272025-05-30T21:40:05.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 84 reason = Timeout
81282025-05-30T21:40:05.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81292025-05-30T21:40:05.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81302025-05-30T21:40:05.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81312025-05-30T21:40:05.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81322025-05-30T21:40:05.946ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81332025-05-30T21:40:05.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81342025-05-30T21:40:05.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
81352025-05-30T21:40:05.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81362025-05-30T21:40:05.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81372025-05-30T21:40:05.950ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81382025-05-30T21:40:05.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81392025-05-30T21:40:05.952ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81402025-05-30T21:40:05.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
81412025-05-30T21:40:05.955ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81422025-05-30T21:40:05.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81432025-05-30T21:40:05.966ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
81442025-05-30T21:40:05.966ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.143755ms iteration = 84 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
81452025-05-30T21:40:05.977ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.95183ms iteration = 84 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
81462025-05-30T21:40:08.779ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 352 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 99bcbc3e-ddef-4040-8182-476c59f22449 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
81472025-05-30T21:40:15.942ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 15 reason = Timeout
81482025-05-30T21:40:15.942ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
81492025-05-30T21:40:15.942ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
81502025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
81512025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 85 reason = Timeout
81522025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81532025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 29 reason = Timeout
81542025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 15 reason = Timeout
81552025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
81562025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
81572025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 15 reason = Timeout
81582025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
81592025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 17 reason = Timeout
81602025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 15 reason = Timeout
81612025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 15 reason = Timeout
81622025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
81632025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
81642025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 29 reason = Timeout
81652025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
81662025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 15 reason = Timeout
81672025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
81682025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
81692025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 31 reason = Timeout
81702025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
81712025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 29 reason = Timeout
81722025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
81732025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 29 reason = Timeout
81742025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
81752025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 29 reason = Timeout
81762025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
81772025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81782025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 29 reason = Timeout
81792025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 29 reason = Timeout
81802025-05-30T21:40:15.943ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
81812025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81822025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
81832025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 15 reason = Timeout
81842025-05-30T21:40:15.943ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
81852025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 29 reason = Timeout
81862025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
81872025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
81882025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 15 reason = Timeout
81892025-05-30T21:40:15.943ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-30 21:30:15.943532429 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
81902025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
81912025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 29 reason = Timeout
81922025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
81932025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 29 reason = Timeout
81942025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
81952025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 29 reason = Timeout
81962025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
81972025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 29 reason = Timeout
81982025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
81992025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 15 reason = Timeout
82002025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
82012025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 29 reason = Timeout
82022025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82032025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 15 reason = Timeout
82042025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
82052025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 29 reason = Timeout
82062025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 15 reason = Timeout
82072025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 85 reason = Timeout
82082025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 29 reason = Timeout
82092025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
82102025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82112025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
82122025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 29 reason = Timeout
82132025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
82142025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 15 reason = Timeout
82152025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
82162025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 16 reason = Timeout
82172025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 17 reason = Timeout
82182025-05-30T21:40:15.943ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e
82192025-05-30T21:40:15.943ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.84\u{b5}s iteration = 16 status = {"enabled":false,"target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e"}
82202025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
82212025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
82222025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
82232025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
82242025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
82252025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
82262025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
82272025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
82282025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
82292025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
82302025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
82312025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
82322025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
82332025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
82342025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
82352025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82362025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
82372025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
82382025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
82392025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
82402025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
82412025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
82422025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
82432025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
82442025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
82452025-05-30T21:40:15.944ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2a161087-f464-41f8-b020-0f9e83fd5351.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("5c275d18-0bbd-414f-a1ca-4f5fb261fa3a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1950, tv_nsec: 878821547 } })
82462025-05-30T21:40:15.944ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0413f6b4-86a1-42a1-919a-e8f42ff9f638.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("acb1452d-0a02-4b36-a462-c8ce2f96a488.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fa7014ba-cffd-46e9-a837-03451be0348b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:2::1))) }], valid_until: Instant { tv_sec: 1950, tv_nsec: 878826588 } })
82472025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
82482025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
82492025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
82502025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 958.702\u{b5}s iteration = 15 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
82512025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } dns_group = internal
82522025-05-30T21:40:15.944ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.002402ms iteration = 15 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353"]}
82532025-05-30T21:40:15.945ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82542025-05-30T21:40:15.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82552025-05-30T21:40:15.947ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82562025-05-30T21:40:15.948ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82572025-05-30T21:40:15.950ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
82582025-05-30T21:40:15.950ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
82592025-05-30T21:40:15.950ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 6.395934ms iteration = 15 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
82602025-05-30T21:40:15.950ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
82612025-05-30T21:40:15.950ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
82622025-05-30T21:40:15.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
82632025-05-30T21:40:15.951ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
82642025-05-30T21:40:15.951ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
82652025-05-30T21:40:15.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
82662025-05-30T21:40:15.951ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
82672025-05-30T21:40:15.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
82682025-05-30T21:40:15.952ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 9.374028ms iteration = 15 status = {"errors":[],"found_port_ok":[]}
82692025-05-30T21:40:15.952ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 9.242156ms iteration = 29 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
82702025-05-30T21:40:15.953ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82712025-05-30T21:40:15.954ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82722025-05-30T21:40:15.955ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1950, tv_nsec: 890496002 } })
82732025-05-30T21:40:15.956ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179
82742025-05-30T21:40:15.956ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
82752025-05-30T21:40:15.956ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
82762025-05-30T21:40:15.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
82772025-05-30T21:40:15.957ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
82782025-05-30T21:40:15.957ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
82792025-05-30T21:40:15.957ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
82802025-05-30T21:40:15.959ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-30 21:30:15.943532429 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
82812025-05-30T21:40:15.959ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 16.054394ms iteration = 15 status = {"expiration":"2025-05-30T21:30:15.943532429Z","pruned":[]}
82822025-05-30T21:40:15.959ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82832025-05-30T21:40:15.961ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82842025-05-30T21:40:15.962ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
82852025-05-30T21:40:15.962ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82862025-05-30T21:40:15.962ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82872025-05-30T21:40:15.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82882025-05-30T21:40:15.963ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:202 num_receivers = 0
82892025-05-30T21:40:15.963ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 19.390563ms iteration = 15 status = {"by_rx":{},"error":null}
82902025-05-30T21:40:15.963ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
82912025-05-30T21:40:15.963ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
82922025-05-30T21:40:15.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
82932025-05-30T21:40:15.966ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
82942025-05-30T21:40:15.969ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 26.364573ms iteration = 85 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
82952025-05-30T21:40:15.970ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
82962025-05-30T21:40:15.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
82972025-05-30T21:40:15.972ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82982025-05-30T21:40:15.973ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82992025-05-30T21:40:15.973ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1950, tv_nsec: 908171354 } })
83002025-05-30T21:40:15.973ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1179
83012025-05-30T21:40:15.976ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
83022025-05-30T21:40:15.976ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83032025-05-30T21:40:15.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83042025-05-30T21:40:15.976ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
83052025-05-30T21:40:15.977ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
83062025-05-30T21:40:15.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
83072025-05-30T21:40:15.979ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83082025-05-30T21:40:15.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83092025-05-30T21:40:15.979ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83102025-05-30T21:40:15.984ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
83112025-05-30T21:40:15.984ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
83122025-05-30T21:40:15.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
83132025-05-30T21:40:15.985ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83142025-05-30T21:40:15.987ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 43.815413ms iteration = 15 status = {"dispatched":[],"errors":[],"glob_version":"144.0.0","globs_reprocessed":{},"no_receivers":[]}
83152025-05-30T21:40:15.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
83162025-05-30T21:40:15.988ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83172025-05-30T21:40:15.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83182025-05-30T21:40:15.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83192025-05-30T21:40:15.991ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83202025-05-30T21:40:15.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83212025-05-30T21:40:15.992ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
83222025-05-30T21:40:15.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83232025-05-30T21:40:15.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83242025-05-30T21:40:15.994ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83252025-05-30T21:40:15.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
83262025-05-30T21:40:15.999ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83272025-05-30T21:40:15.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83282025-05-30T21:40:16.000ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 56.787913ms iteration = 29 status = {"errors":[],"garbage_collect_requested":[]}
83292025-05-30T21:40:16.005ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
83302025-05-30T21:40:16.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
83312025-05-30T21:40:16.008ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83322025-05-30T21:40:16.008ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
83332025-05-30T21:40:16.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
83342025-05-30T21:40:16.011ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
83352025-05-30T21:40:16.011ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83362025-05-30T21:40:16.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83372025-05-30T21:40:16.012ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 68.998662ms iteration = 29 status = {"errors":[],"finish_invoked_ok":[]}
83382025-05-30T21:40:16.013ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83392025-05-30T21:40:16.013ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 70.375628ms iteration = 29 status = {"errors":[],"requests_created_ok":[]}
83402025-05-30T21:40:16.015ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
83412025-05-30T21:40:16.015ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-5d926227-6a52-4938-b74d-21ba39bce2f9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1950, tv_nsec: 950037885 } })
83422025-05-30T21:40:16.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
83432025-05-30T21:40:16.015ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
83442025-05-30T21:40:16.015ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 72.108778ms iteration = 85 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
83452025-05-30T21:40:16.015ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
83462025-05-30T21:40:16.016ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
83472025-05-30T21:40:16.016ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
83482025-05-30T21:40:16.017ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
83492025-05-30T21:40:16.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
83502025-05-30T21:40:16.018ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
83512025-05-30T21:40:16.018ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
83522025-05-30T21:40:16.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83532025-05-30T21:40:16.019ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
83542025-05-30T21:40:16.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
83552025-05-30T21:40:16.021ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
83562025-05-30T21:40:16.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83572025-05-30T21:40:16.021ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83582025-05-30T21:40:16.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
83592025-05-30T21:40:16.024ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
83602025-05-30T21:40:16.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
83612025-05-30T21:40:16.027ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
83622025-05-30T21:40:16.027ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
83632025-05-30T21:40:16.027ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 84.341858ms iteration = 15 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
83642025-05-30T21:40:16.030ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
83652025-05-30T21:40:16.030ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
83662025-05-30T21:40:16.031ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
83672025-05-30T21:40:16.031ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
83682025-05-30T21:40:16.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
83692025-05-30T21:40:16.031ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
83702025-05-30T21:40:16.031ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
83712025-05-30T21:40:16.031ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
83722025-05-30T21:40:16.031ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
83732025-05-30T21:40:16.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
83742025-05-30T21:40:16.031ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:547 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
83752025-05-30T21:40:16.031ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 88.182703ms iteration = 29 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
83762025-05-30T21:40:16.032ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83772025-05-30T21:40:16.033ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83782025-05-30T21:40:16.036ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
83792025-05-30T21:40:16.036ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
83802025-05-30T21:40:16.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
83812025-05-30T21:40:16.036ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
83822025-05-30T21:40:16.036ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
83832025-05-30T21:40:16.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
83842025-05-30T21:40:16.039ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
83852025-05-30T21:40:16.039ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
83862025-05-30T21:40:16.039ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
83872025-05-30T21:40:16.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
83882025-05-30T21:40:16.046ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83892025-05-30T21:40:16.049ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
83902025-05-30T21:40:16.050ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
83912025-05-30T21:40:16.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
83922025-05-30T21:40:16.055ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
83932025-05-30T21:40:16.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
83942025-05-30T21:40:16.056ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
83952025-05-30T21:40:16.057ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
83962025-05-30T21:40:16.058ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83972025-05-30T21:40:16.059ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 115.687729ms iteration = 29 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
83982025-05-30T21:40:16.060ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 6809f39a-1ba0-430e-994e-26499f509fac dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
83992025-05-30T21:40:16.060ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
84002025-05-30T21:40:16.060ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-30 21:26:17.226115 UTC dns_group = internal generation = 1
84012025-05-30T21:40:16.060ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 117.904804ms iteration = 15 status = {"generation":1}
84022025-05-30T21:40:16.061ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
84032025-05-30T21:40:16.061ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
84042025-05-30T21:40:16.061ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
84052025-05-30T21:40:16.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
84062025-05-30T21:40:16.064ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 120.626225ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
84072025-05-30T21:40:16.064ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
84082025-05-30T21:40:16.064ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
84092025-05-30T21:40:16.064ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
84102025-05-30T21:40:16.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
84112025-05-30T21:40:16.068ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 124.831554ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
84122025-05-30T21:40:16.069ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
84132025-05-30T21:40:16.073ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84142025-05-30T21:40:16.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84152025-05-30T21:40:16.083ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
84162025-05-30T21:40:16.083ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 139.848796ms iteration = 29 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
84172025-05-30T21:40:16.095ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
84182025-05-30T21:40:16.095ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 0 version = 2
84192025-05-30T21:40:16.095ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
84202025-05-30T21:40:16.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
84212025-05-30T21:40:16.113ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
84222025-05-30T21:40:16.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
84232025-05-30T21:40:16.114ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
84242025-05-30T21:40:16.114ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
84252025-05-30T21:40:16.115ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d1a91a4-45f7-4b08-9d00-d404ce0dc382", "content-length": "26", "date": "Fri, 30 May 2025 21:40:16 GMT"} })
84262025-05-30T21:40:16.115ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
84272025-05-30T21:40:16.117ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
84282025-05-30T21:40:16.117ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
84292025-05-30T21:40:16.117ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 173.332571ms iteration = 29 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
84302025-05-30T21:40:16.118ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
84312025-05-30T21:40:16.120ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
84322025-05-30T21:40:16.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
84332025-05-30T21:40:16.122ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-30T21:26:13.986396Z, time_modified: 2025-05-30T21:26:13.986396Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: a5a913ca-db42-49f6-993f-94a0df8e5c59, db_silo: Silo { identity: SiloIdentity { id: a5a913ca-db42-49f6-993f-94a0df8e5c59, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-30T21:26:18.773108Z, time_modified: 2025-05-30T21:26:18.773108Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9" }] }) }
84342025-05-30T21:40:16.122ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 179.38502ms iteration = 15 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]}},"default_endpoint":{"silo_id":"a5a913ca-db42-49f6-993f-94a0df8e5c59","tls_certs":["3e2cbf1fd89ab3a53d171d1abe55788c552ab5fffe0846d708a7589d586f95d9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
84352025-05-30T21:40:16.134ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1187 zone_address = fd00:1122:3344:101::2
84362025-05-30T21:40:16.134ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
84372025-05-30T21:40:16.135ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d86cac1-28ef-4280-a546-963d9b015e4e", "content-length": "26", "date": "Fri, 30 May 2025 21:40:16 GMT"} })
84382025-05-30T21:40:16.135ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1190 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
84392025-05-30T21:40:16.137ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
84402025-05-30T21:40:16.164ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 221.011588ms iteration = 31 status = {"physical_disks_added":0}
84412025-05-30T21:40:16.207ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1187 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f zone_address = fd00:1122:3344:101::2
84422025-05-30T21:40:16.207ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
84432025-05-30T21:40:16.207ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "77429774-d18d-468b-aaff-2484c3d41a2f", "content-length": "26", "date": "Fri, 30 May 2025 21:40:16 GMT"} })
84442025-05-30T21:40:16.207ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1190 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
84452025-05-30T21:40:16.210ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1224 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
84462025-05-30T21:40:16.222ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
84472025-05-30T21:40:16.253ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
84482025-05-30T21:40:16.253ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "a262a818-1aef-4cdc-a3d0-aafb19f06772", "date": "Fri, 30 May 2025 21:40:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
84492025-05-30T21:40:16.253ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
84502025-05-30T21:40:16.254ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "79230148-7614-4a26-80fb-bc08f213ac19", "date": "Fri, 30 May 2025 21:40:15 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
84512025-05-30T21:40:16.254ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
84522025-05-30T21:40:16.256ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
84532025-05-30T21:40:16.256ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 312.681502ms iteration = 17 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
84542025-05-30T21:40:16.266ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
84552025-05-30T21:40:16.266ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
84562025-05-30T21:40:16.266ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "31b0b051-690e-48f8-a794-04988f826060", "content-length": "372", "date": "Fri, 30 May 2025 21:40:15 GMT"} })
84572025-05-30T21:40:16.266ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84582025-05-30T21:40:16.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84592025-05-30T21:40:16.268ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84602025-05-30T21:40:16.268ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84612025-05-30T21:40:16.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84622025-05-30T21:40:16.271ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
84632025-05-30T21:40:16.271ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
84642025-05-30T21:40:16.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
84652025-05-30T21:40:16.272ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 329.343873ms iteration = 29 status = {}
84662025-05-30T21:40:16.273ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84672025-05-30T21:40:16.274ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 9c2750e7-4128-452a-bf8a-cdcce54c8e3f dns_zone_name = oxide.test found_so_far = 1 version = 2
84682025-05-30T21:40:16.274ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
84692025-05-30T21:40:16.274ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-30 21:26:18.957109 UTC dns_group = external generation = 2
84702025-05-30T21:40:16.274ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 330.822559ms iteration = 15 status = {"generation":2}
84712025-05-30T21:40:16.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 5d926227-6a52-4938-b74d-21ba39bce2f9, lookup_type: ById(5d926227-6a52-4938-b74d-21ba39bce2f9) } result = Ok(())
84722025-05-30T21:40:16.276ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84732025-05-30T21:40:16.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84742025-05-30T21:40:16.278ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 334.755105ms iteration = 29 status = 0
84752025-05-30T21:40:16.281ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
84762025-05-30T21:40:16.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
84772025-05-30T21:40:16.283ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 340.050776ms iteration = 29 status = null
84782025-05-30T21:40:16.302ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
84792025-05-30T21:40:16.303ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "2fa39809-aa53-4137-b2da-9450fd711c3b", "date": "Fri, 30 May 2025 21:40:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
84802025-05-30T21:40:16.303ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
84812025-05-30T21:40:16.305ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "9fba78d1-0d8a-4262-9b5a-39ab03eaa126", "date": "Fri, 30 May 2025 21:40:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
84822025-05-30T21:40:16.305ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
84832025-05-30T21:40:16.306ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "db111337-14dc-48e4-baac-44db31560d1b", "date": "Fri, 30 May 2025 21:40:16 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
84842025-05-30T21:40:16.306ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
84852025-05-30T21:40:16.316ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353] }
84862025-05-30T21:40:16.316ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 373.017714ms iteration = 17 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
84872025-05-30T21:40:16.343ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
84882025-05-30T21:40:16.343ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
84892025-05-30T21:40:16.343ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "73b4f879-427f-4dca-9185-a3ecca158986", "date": "Fri, 30 May 2025 21:40:15 GMT"} })
84902025-05-30T21:40:16.343ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84912025-05-30T21:40:16.343ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84922025-05-30T21:40:16.353ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84932025-05-30T21:40:16.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84942025-05-30T21:40:16.357ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84952025-05-30T21:40:16.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84962025-05-30T21:40:16.360ZINFOnexus (BackgroundTasks): VPC route manager sled 5d926227-6a52-4938-b74d-21ba39bce2f9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
84972025-05-30T21:40:16.360ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
84982025-05-30T21:40:16.360ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "f6799df6-c9f5-44b0-be8b-f772999dd49c", "date": "Fri, 30 May 2025 21:40:15 GMT"} })
84992025-05-30T21:40:16.363ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 419.827502ms iteration = 29 status = {}
85002025-05-30T21:40:16.394ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85012025-05-30T21:40:16.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85022025-05-30T21:40:16.396ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 869f9b91-73b5-44e6-9d7b-569223b56fc2, rack_id: ee56532e-87c3-4f14-ab77-c7be0761ee5f, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b27a3244-36d1-47e4-812b-1cffe3a42ec3) } rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
85032025-05-30T21:40:16.396ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85042025-05-30T21:40:16.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85052025-05-30T21:40:16.409ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
85062025-05-30T21:40:16.409ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b27a3244-36d1-47e4-812b-1cffe3a42ec3
85072025-05-30T21:40:16.409ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
85082025-05-30T21:40:16.409ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
85092025-05-30T21:40:16.409ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
85102025-05-30T21:40:16.409ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
85112025-05-30T21:40:16.409ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
85122025-05-30T21:40:16.409ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
85132025-05-30T21:40:16.409ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
85142025-05-30T21:40:16.409ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
85152025-05-30T21:40:16.409ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
85162025-05-30T21:40:16.410ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f
85172025-05-30T21:40:16.410ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85182025-05-30T21:40:16.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
85192025-05-30T21:40:16.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85202025-05-30T21:40:16.411ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85212025-05-30T21:40:16.412ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch_location = Switch0
85222025-05-30T21:40:16.412ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
85232025-05-30T21:40:16.413ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
85242025-05-30T21:40:16.413ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
85252025-05-30T21:40:16.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
85262025-05-30T21:40:16.415ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85272025-05-30T21:40:16.417ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
85282025-05-30T21:40:16.417ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
85292025-05-30T21:40:16.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
85302025-05-30T21:40:16.423ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
85312025-05-30T21:40:16.423ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 479.846832ms iteration = 15 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
85322025-05-30T21:40:16.518ZDEBGnexus (BackgroundTasks): client request SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
85332025-05-30T21:40:16.524ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "25db4f9e-3f0c-41bd-9e83-efad02fb3d66", "date": "Fri, 30 May 2025 21:40:15 GMT"} })
85342025-05-30T21:40:16.524ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f switch = Switch0
85352025-05-30T21:40:16.524ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
85362025-05-30T21:40:16.524ZDEBGnexus (BackgroundTasks): client response SledAgent = 5d926227-6a52-4938-b74d-21ba39bce2f9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "60070f31-0d36-491b-a4cf-6f73885c0200", "content-length": "691", "date": "Fri, 30 May 2025 21:40:15 GMT"} })
85372025-05-30T21:40:16.524ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
85382025-05-30T21:40:16.524ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
85392025-05-30T21:40:16.524ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
85402025-05-30T21:40:16.524ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ee56532e-87c3-4f14-ab77-c7be0761ee5f result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c50ae9f0-4c3f-4b52-a543-38d0ee06b892", "content-length": "682", "date": "Fri, 30 May 2025 21:40:16 GMT"} })
85412025-05-30T21:40:16.524ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85422025-05-30T21:40:16.524ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 581.278226ms iteration = 29 status = {}
85432025-05-30T21:40:16.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85442025-05-30T21:40:16.528ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85452025-05-30T21:40:16.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85462025-05-30T21:40:16.532ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85472025-05-30T21:40:16.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85482025-05-30T21:40:16.534ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85492025-05-30T21:40:16.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85502025-05-30T21:40:16.542ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 598.670857ms iteration = 29 status = {}
85512025-05-30T21:40:18.779ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 316 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 61d36cd5-3a71-45b9-af5e-2e44903f04f3 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2
85522025-05-30T21:40:25.943ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 86 reason = Timeout
85532025-05-30T21:40:25.943ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 86 reason = Timeout
85542025-05-30T21:40:25.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85552025-05-30T21:40:25.943ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85562025-05-30T21:40:25.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85572025-05-30T21:40:25.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85582025-05-30T21:40:25.946ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85592025-05-30T21:40:25.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85602025-05-30T21:40:25.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85612025-05-30T21:40:25.948ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.226569ms iteration = 86 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
85622025-05-30T21:40:25.949ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
85632025-05-30T21:40:25.949ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85642025-05-30T21:40:25.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85652025-05-30T21:40:25.951ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85662025-05-30T21:40:25.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85672025-05-30T21:40:25.953ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85682025-05-30T21:40:25.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 32998c56-2cbd-4c39-afad-1a99febbe84e, lookup_type: ById(32998c56-2cbd-4c39-afad-1a99febbe84e) } result = Ok(())
85692025-05-30T21:40:25.955ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85702025-05-30T21:40:25.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85712025-05-30T21:40:25.964ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e original_time_created = 2025-05-30 21:26:07.587606 UTC target_id = 32998c56-2cbd-4c39-afad-1a99febbe84e time_created = 2025-05-30 21:26:07.587606 UTC
85722025-05-30T21:40:25.964ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.496116ms iteration = 86 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"32998c56-2cbd-4c39-afad-1a99febbe84e","time_created":"2025-05-30T21:26:07.587606Z"}
85732025-05-30T21:40:28.769ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:47265
85742025-05-30T21:40:28.769ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47265 req_id = d7f9b399-9cab-4de2-bec7-586af83ca987 roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
85752025-05-30T21:40:28.770ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47265 req_id = d7f9b399-9cab-4de2-bec7-586af83ca987 resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
85762025-05-30T21:40:28.773ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3393 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47265 req_id = d7f9b399-9cab-4de2-bec7-586af83ca987 response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?sort_by=id_ascending
85772025-05-30T21:40:28.773ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47265 req_id = 66e00897-c15d-48ab-b596-23481e70db09 roles = RoleSet { roles: {} } uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
85782025-05-30T21:40:28.773ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47265 req_id = 66e00897-c15d-48ab-b596-23481e70db09 resource = Database result = Ok(()) uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
85792025-05-30T21:40:28.775ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2469 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47265 req_id = 66e00897-c15d-48ab-b596-23481e70db09 response_code = 200 uri = /metrics/collectors/bc707af4-d9a3-402f-b2b6-642b29342fa8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmYTIzZmQ2OS0wNjRmLTRlYzktYWJiYi1hODc0MDg0MmQ3NjMifX0%3D
85802025-05-30T21:40:28.778ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 296 local_addr = [fd00:1122:3344:101::b]:43098 method = GET remote_addr = [fd00:1122:3344:101::d]:63628 req_id = 4be554d6-c215-4757-816c-247130a05be1 response_code = 200 uri = /bb2ce2e9-85b4-4cff-aedf-120e3ffadec2