Max level shown:
12025-03-25T23:00:03.857ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 416 local_addr = [fd00:1122:3344:101::b]:56059 method = GET remote_addr = [fd00:1122:3344:101::d]:46144 req_id = 390a547d-2b9c-4bef-a5ff-c37a56bd19f6 response_code = 200 uri = /d5542f9d-430d-4b19-a4dc-31cd924093fb
22025-03-25T23:00:12.157ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 22 reason = Timeout
32025-03-25T23:00:12.157ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42025-03-25T23:00:12.157ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 22 reason = Timeout
52025-03-25T23:00:12.157ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 22 reason = Timeout
62025-03-25T23:00:12.157ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
72025-03-25T23:00:12.157ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 22 reason = Timeout
82025-03-25T23:00:12.157ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
92025-03-25T23:00:12.157ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
102025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 22 reason = Timeout
112025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
122025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 22 reason = Timeout
132025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 24 reason = Timeout
142025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 64 reason = Timeout
152025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
162025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
172025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
182025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 22 reason = Timeout
192025-03-25T23:00:12.158ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
202025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 22 reason = Timeout
212025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
222025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
232025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 22 reason = Timeout
242025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
252025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 22 reason = Timeout
262025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 22 reason = Timeout
272025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
282025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
292025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 22 reason = Timeout
302025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
312025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
322025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 64 reason = Timeout
332025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
342025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 22 reason = Timeout
352025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
362025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 22 reason = Timeout
372025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
382025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 22 reason = Timeout
392025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 22 reason = Timeout
402025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
412025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
422025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
432025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
442025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
452025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
462025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
472025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
482025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
492025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
502025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
512025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
522025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
532025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
542025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
552025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
562025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
572025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
582025-03-25T23:00:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
592025-03-25T23:00:12.162ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 4.541202ms iteration = 22 status = {"errors":[],"garbage_collect_requested":[]}
602025-03-25T23:00:12.162ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.378669ms iteration = 22 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
612025-03-25T23:00:12.162ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
622025-03-25T23:00:12.163ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
632025-03-25T23:00:12.164ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
642025-03-25T23:00:12.168ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
652025-03-25T23:00:12.168ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
662025-03-25T23:00:12.168ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
672025-03-25T23:00:12.168ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
682025-03-25T23:00:12.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
692025-03-25T23:00:12.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
702025-03-25T23:00:12.168ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
712025-03-25T23:00:12.168ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
722025-03-25T23:00:12.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
732025-03-25T23:00:12.171ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1785, tv_nsec: 617531950 } })
742025-03-25T23:00:12.171ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
752025-03-25T23:00:12.171ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
762025-03-25T23:00:12.172ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
772025-03-25T23:00:12.172ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1785, tv_nsec: 618758818 } })
782025-03-25T23:00:12.172ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
792025-03-25T23:00:12.172ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
802025-03-25T23:00:12.174ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
812025-03-25T23:00:12.177ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
822025-03-25T23:00:12.177ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
832025-03-25T23:00:12.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
842025-03-25T23:00:12.178ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
852025-03-25T23:00:12.178ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
862025-03-25T23:00:12.178ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
872025-03-25T23:00:12.179ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
882025-03-25T23:00:12.179ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
892025-03-25T23:00:12.183ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 25.412663ms iteration = 22 status = {"errors":[],"requests_created_ok":[]}
902025-03-25T23:00:12.186ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
912025-03-25T23:00:12.187ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
922025-03-25T23:00:12.188ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
932025-03-25T23:00:12.190ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
942025-03-25T23:00:12.190ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
952025-03-25T23:00:12.190ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
962025-03-25T23:00:12.190ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
972025-03-25T23:00:12.194ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
982025-03-25T23:00:12.194ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
992025-03-25T23:00:12.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1002025-03-25T23:00:12.197ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1012025-03-25T23:00:12.198ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1022025-03-25T23:00:12.199ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 41.035705ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
1032025-03-25T23:00:12.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1042025-03-25T23:00:12.201ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1052025-03-25T23:00:12.201ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1062025-03-25T23:00:12.201ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1072025-03-25T23:00:12.205ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1082025-03-25T23:00:12.206ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1092025-03-25T23:00:12.207ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1102025-03-25T23:00:12.207ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1112025-03-25T23:00:12.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1122025-03-25T23:00:12.212ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 5d9d6f43-fd71-4256-b06d-1ed4c87e191f, lookup_type: ById(5d9d6f43-fd71-4256-b06d-1ed4c87e191f) } result = Ok(())
1132025-03-25T23:00:12.212ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1142025-03-25T23:00:12.212ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1152025-03-25T23:00:12.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1162025-03-25T23:00:12.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1172025-03-25T23:00:12.222ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1785, tv_nsec: 668631672 } })
1182025-03-25T23:00:12.222ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619
1192025-03-25T23:00:12.229ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 71.148894ms iteration = 22 status = {"errors":[],"finish_invoked_ok":[]}
1202025-03-25T23:00:12.243ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1212025-03-25T23:00:12.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1222025-03-25T23:00:12.249ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1232025-03-25T23:00:12.250ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1242025-03-25T23:00:12.254ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1252025-03-25T23:00:12.254ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
1262025-03-25T23:00:12.254ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1272025-03-25T23:00:12.254ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1282025-03-25T23:00:12.255ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
1292025-03-25T23:00:12.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1302025-03-25T23:00:12.264ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1312025-03-25T23:00:12.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1322025-03-25T23:00:12.267ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1332025-03-25T23:00:12.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1342025-03-25T23:00:12.282ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 124.345352ms iteration = 24 status = {"physical_disks_added":0}
1352025-03-25T23:00:12.284ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
1362025-03-25T23:00:12.284ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1372025-03-25T23:00:12.285ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d2b9360-01d5-4e35-9c5f-cb70d384a009", "content-length": "26", "date": "Tue, 25 Mar 2025 23:00:12 GMT"} })
1382025-03-25T23:00:12.285ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1392025-03-25T23:00:12.287ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1402025-03-25T23:00:12.300ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
1412025-03-25T23:00:12.300ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1422025-03-25T23:00:12.300ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a07591d-20e7-47f6-ba5a-0291cdf983f2", "content-length": "26", "date": "Tue, 25 Mar 2025 23:00:12 GMT"} })
1432025-03-25T23:00:12.300ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1442025-03-25T23:00:12.302ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1452025-03-25T23:00:12.349ZINFOnexus (BackgroundTasks): VPC route manager sled 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
1462025-03-25T23:00:12.349ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
1472025-03-25T23:00:12.349ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "c74a9cf6-830c-4c26-99f9-64109eccf592", "content-length": "585", "date": "Tue, 25 Mar 2025 23:00:12 GMT"} })
1482025-03-25T23:00:12.350ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1492025-03-25T23:00:12.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1502025-03-25T23:00:12.354ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 196.08504ms iteration = 64 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
1512025-03-25T23:00:12.359ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1522025-03-25T23:00:12.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1532025-03-25T23:00:12.360ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f original_time_created = 2025-03-25 22:49:35.383803 UTC target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f time_created = 2025-03-25 22:49:35.383803 UTC
1542025-03-25T23:00:12.360ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 202.460684ms iteration = 64 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f","time_created":"2025-03-25T22:49:35.383803Z"}
1552025-03-25T23:00:12.363ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = instance_watcher body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vmms/3214a533-86bc-409a-9c53-0114031452e2/state
1562025-03-25T23:00:12.366ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = instance_watcher result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vmms/3214a533-86bc-409a-9c53-0114031452e2/state", status: 200, headers: {"content-type": "application/json", "x-request-id": "1599e6e9-28b3-4957-9982-d5fba48463ae", "content-length": "130", "date": "Tue, 25 Mar 2025 23:00:12 GMT"} })
1572025-03-25T23:00:12.366ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1582025-03-25T23:00:12.367ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1592025-03-25T23:00:12.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1602025-03-25T23:00:12.369ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1612025-03-25T23:00:12.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1622025-03-25T23:00:12.371ZDEBGnexus (BackgroundTasks): updating instance state background_task = instance_watcher instance_id = 42a2133b-3721-4ce4-8ffe-ac478e6f81cd sled_id = 9af3e790-030d-4974-9b9e-c35f829f8643 state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-03-25T23:00:12.015112346Z }, migration_in: None, migration_out: None } vmm_id = 3214a533-86bc-409a-9c53-0114031452e2
1632025-03-25T23:00:12.371ZINFOnexus (BackgroundTasks): received new VMM runtime state from sled agent background_task = instance_watcher file = nexus/src/app/instance.rs:2120 instance_id = 42a2133b-3721-4ce4-8ffe-ac478e6f81cd migration_state = Migrations { migration_in: None, migration_out: None } propolis_id = 3214a533-86bc-409a-9c53-0114031452e2 sled_id = 9af3e790-030d-4974-9b9e-c35f829f8643 vmm_id = 3214a533-86bc-409a-9c53-0114031452e2 vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-03-25T23:00:12.015112346Z }
1642025-03-25T23:00:12.371ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 42a2133b-3721-4ce4-8ffe-ac478e6f81cd roles = RoleSet { roles: {} } sled_id = 9af3e790-030d-4974-9b9e-c35f829f8643 vmm_id = 3214a533-86bc-409a-9c53-0114031452e2
1652025-03-25T23:00:12.371ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
1662025-03-25T23:00:12.371ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1672025-03-25T23:00:12.371ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 zone_address = fd00:1122:3344:101::2
1682025-03-25T23:00:12.371ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1692025-03-25T23:00:12.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 42a2133b-3721-4ce4-8ffe-ac478e6f81cd resource = Database result = Ok(()) sled_id = 9af3e790-030d-4974-9b9e-c35f829f8643 vmm_id = 3214a533-86bc-409a-9c53-0114031452e2
1702025-03-25T23:00:12.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1712025-03-25T23:00:12.372ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ebac610-d163-4d94-8b84-8448bc9ce266", "content-length": "26", "date": "Tue, 25 Mar 2025 23:00:12 GMT"} })
1722025-03-25T23:00:12.372ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1732025-03-25T23:00:12.374ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619
1742025-03-25T23:00:12.374ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1752025-03-25T23:00:12.377ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
1762025-03-25T23:00:12.377ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1772025-03-25T23:00:12.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1782025-03-25T23:00:12.378ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1792025-03-25T23:00:12.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1802025-03-25T23:00:12.382ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1812025-03-25T23:00:12.383ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1822025-03-25T23:00:12.385ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1832025-03-25T23:00:12.388ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 9af3e790-030d-4974-9b9e-c35f829f8643, lookup_type: ById(9af3e790-030d-4974-9b9e-c35f829f8643) } result = Ok(())
1842025-03-25T23:00:12.389ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1852025-03-25T23:00:12.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1862025-03-25T23:00:12.391ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 233.533935ms iteration = 22 status = 0
1872025-03-25T23:00:12.403ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
1882025-03-25T23:00:12.403ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7ca8ba4-a8eb-4722-9093-6c669d31949f", "content-length": "1", "date": "Tue, 25 Mar 2025 23:00:11 GMT"} })
1892025-03-25T23:00:12.404ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 246.610279ms iteration = 22 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
1902025-03-25T23:00:12.405ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
1912025-03-25T23:00:12.405ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1922025-03-25T23:00:12.407ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 249.406422ms iteration = 22 status = null
1932025-03-25T23:00:12.407ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
1942025-03-25T23:00:12.407ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 249.700359ms iteration = 22 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
1952025-03-25T23:00:12.408ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1962025-03-25T23:00:12.408ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
1972025-03-25T23:00:12.408ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1982025-03-25T23:00:12.408ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1992025-03-25T23:00:12.409ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 251.731954ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2002025-03-25T23:00:12.424ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 1 total_failed = 0 total_incomplete = 0 total_instances = 1 update_sagas_queued = 0
2012025-03-25T23:00:12.426ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 268.721938ms iteration = 22 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{"running":1},"pruned_instances":0,"total_instances":1,"update_sagas_queued":0}
2022025-03-25T23:00:12.426ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {5518013d-08ec-4edf-8ef0-a1772322635d: {10.151.1.176: {5443e716-1a40-4091-8e79-6b44910106e0}, 10.151.1.175: {5443e716-1a40-4091-8e79-6b44910106e0}}} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
2032025-03-25T23:00:12.426ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
2042025-03-25T23:00:12.431ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 274.013357ms iteration = 22 status = {}
2052025-03-25T23:00:12.450ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2062025-03-25T23:00:12.450ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
2072025-03-25T23:00:12.450ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 292.37181ms iteration = 22 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
2082025-03-25T23:00:12.535ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
2092025-03-25T23:00:12.535ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "29d639cb-d210-4556-9e13-2fcde02662f7", "date": "Tue, 25 Mar 2025 23:00:12 GMT"} })
2102025-03-25T23:00:12.535ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2112025-03-25T23:00:12.536ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "175133b6-ab16-4997-ad93-5b1fe9c60ae2", "content-length": "811", "date": "Tue, 25 Mar 2025 23:00:12 GMT"} })
2122025-03-25T23:00:12.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2132025-03-25T23:00:12.536ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {VirtualNetworkInterfaceHost { physical_host_ip: fd00:1122:3344:101::1, virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), vni: Vni(14194516) }} sled = buskin
2142025-03-25T23:00:12.536ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
2152025-03-25T23:00:12.536ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
2162025-03-25T23:00:12.538ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 380.260791ms iteration = 22 status = {}
2172025-03-25T23:00:12.539ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2182025-03-25T23:00:12.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2192025-03-25T23:00:12.542ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2202025-03-25T23:00:12.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2212025-03-25T23:00:12.546ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2222025-03-25T23:00:12.547ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2232025-03-25T23:00:12.549ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2242025-03-25T23:00:12.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2252025-03-25T23:00:12.553ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2262025-03-25T23:00:12.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2272025-03-25T23:00:12.557ZINFOnexus (BackgroundTasks): VPC route manager sled 9af3e790-030d-4974-9b9e-c35f829f8643 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
2282025-03-25T23:00:12.557ZINFOnexus (BackgroundTasks): VPC route manager sled 9af3e790-030d-4974-9b9e-c35f829f8643 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
2292025-03-25T23:00:12.557ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2302025-03-25T23:00:12.565ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2312025-03-25T23:00:12.565ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2322025-03-25T23:00:12.568ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 25f1877d-a731-48fe-8967-16ef2fb00f3b, rack_id: 3ef3cc6b-3ffd-47a3-99ef-c57b25776619, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(dcb3f9bf-5ea7-47e3-b699-6c3a8c7459c5) } rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch_location = Switch0
2332025-03-25T23:00:12.568ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2342025-03-25T23:00:12.568ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2352025-03-25T23:00:12.580ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
2362025-03-25T23:00:12.580ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "9970bd24-6cb7-4ef4-89f8-cfea5faa5395", "content-length": "134", "date": "Tue, 25 Mar 2025 23:00:11 GMT"} })
2372025-03-25T23:00:12.580ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
2382025-03-25T23:00:12.580ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = dcb3f9bf-5ea7-47e3-b699-6c3a8c7459c5
2392025-03-25T23:00:12.581ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
2402025-03-25T23:00:12.581ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
2412025-03-25T23:00:12.581ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2422025-03-25T23:00:12.581ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2432025-03-25T23:00:12.581ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2442025-03-25T23:00:12.581ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2452025-03-25T23:00:12.581ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2462025-03-25T23:00:12.581ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2472025-03-25T23:00:12.581ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619
2482025-03-25T23:00:12.581ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2492025-03-25T23:00:12.582ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2502025-03-25T23:00:12.583ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
2512025-03-25T23:00:12.584ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "e740b9ca-b33b-4d86-96f0-0ab98dcc9a4a", "content-length": "2", "date": "Tue, 25 Mar 2025 23:00:11 GMT"} })
2522025-03-25T23:00:12.584ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
2532025-03-25T23:00:12.584ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "21fb6c29-32fd-4e3a-a915-8787d5b425fb", "content-length": "52", "date": "Tue, 25 Mar 2025 23:00:11 GMT"} })
2542025-03-25T23:00:12.584ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch_location = Switch0
2552025-03-25T23:00:12.584ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
2562025-03-25T23:00:12.697ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "f3c510e3-23c1-4b64-a245-3f36639d6be4", "date": "Tue, 25 Mar 2025 23:00:12 GMT"} })
2572025-03-25T23:00:12.698ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 540.822041ms iteration = 22 status = {}
2582025-03-25T23:00:12.830ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "332c2da5-f3be-4721-9df1-2f9ca07833b9", "date": "Tue, 25 Mar 2025 23:00:12 GMT"} })
2592025-03-25T23:00:12.830ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch = Switch0
2602025-03-25T23:00:12.830ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
2612025-03-25T23:00:12.830ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "809e24f6-9335-44e3-8e65-b2b8bb782e32", "content-length": "682", "date": "Tue, 25 Mar 2025 23:00:11 GMT"} })
2622025-03-25T23:00:12.830ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2632025-03-25T23:00:12.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2642025-03-25T23:00:12.835ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2652025-03-25T23:00:12.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2662025-03-25T23:00:12.840ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2672025-03-25T23:00:12.840ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2682025-03-25T23:00:12.842ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2692025-03-25T23:00:12.843ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2702025-03-25T23:00:12.853ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 694.918515ms iteration = 22 status = {}
2712025-03-25T23:00:13.860ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2808 local_addr = [fd00:1122:3344:101::b]:56059 method = GET remote_addr = [fd00:1122:3344:101::d]:46144 req_id = 6ae85e3b-9ee5-4a18-a303-f5d24e55ed9b response_code = 200 uri = /d5542f9d-430d-4b19-a4dc-31cd924093fb
2722025-03-25T23:00:22.157ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 65 reason = Timeout
2732025-03-25T23:00:22.157ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 65 reason = Timeout
2742025-03-25T23:00:22.157ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2752025-03-25T23:00:22.157ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2762025-03-25T23:00:22.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2772025-03-25T23:00:22.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2782025-03-25T23:00:22.160ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2792025-03-25T23:00:22.160ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2802025-03-25T23:00:22.164ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2812025-03-25T23:00:22.167ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2822025-03-25T23:00:22.167ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2832025-03-25T23:00:22.167ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2842025-03-25T23:00:22.169ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2852025-03-25T23:00:22.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2862025-03-25T23:00:22.170ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2872025-03-25T23:00:22.173ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 5d9d6f43-fd71-4256-b06d-1ed4c87e191f, lookup_type: ById(5d9d6f43-fd71-4256-b06d-1ed4c87e191f) } result = Ok(())
2882025-03-25T23:00:22.173ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2892025-03-25T23:00:22.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2902025-03-25T23:00:22.183ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f original_time_created = 2025-03-25 22:49:35.383803 UTC target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f time_created = 2025-03-25 22:49:35.383803 UTC
2912025-03-25T23:00:22.183ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.741711ms iteration = 65 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f","time_created":"2025-03-25T22:49:35.383803Z"}
2922025-03-25T23:00:22.199ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 41.71195ms iteration = 65 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
2932025-03-25T23:00:23.840ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:33395
2942025-03-25T23:00:23.840ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:33395 req_id = d97edec8-2cd8-481c-b658-c0599bda5b51 roles = RoleSet { roles: {} } uri = /metrics/collectors/aa92c5e2-2c4d-4351-b468-f1bc420a118d/producers?sort_by=id_ascending
2952025-03-25T23:00:23.841ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:33395 req_id = d97edec8-2cd8-481c-b658-c0599bda5b51 resource = Database result = Ok(()) uri = /metrics/collectors/aa92c5e2-2c4d-4351-b468-f1bc420a118d/producers?sort_by=id_ascending
2962025-03-25T23:00:23.844ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3407 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:33395 req_id = d97edec8-2cd8-481c-b658-c0599bda5b51 response_code = 200 uri = /metrics/collectors/aa92c5e2-2c4d-4351-b468-f1bc420a118d/producers?sort_by=id_ascending
2972025-03-25T23:00:23.844ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:33395 req_id = 20c9f440-eff3-4cd4-8a61-1978e03fc511 roles = RoleSet { roles: {} } uri = /metrics/collectors/aa92c5e2-2c4d-4351-b468-f1bc420a118d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzQ4MTE5Yy05YTdjLTRiNWQtOGNkOC1mZWRkZjQ3NjU5ZDUifX0%3D
2982025-03-25T23:00:23.844ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:33395 req_id = 20c9f440-eff3-4cd4-8a61-1978e03fc511 resource = Database result = Ok(()) uri = /metrics/collectors/aa92c5e2-2c4d-4351-b468-f1bc420a118d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzQ4MTE5Yy05YTdjLTRiNWQtOGNkOC1mZWRkZjQ3NjU5ZDUifX0%3D
2992025-03-25T23:00:23.846ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2655 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:33395 req_id = 20c9f440-eff3-4cd4-8a61-1978e03fc511 response_code = 200 uri = /metrics/collectors/aa92c5e2-2c4d-4351-b468-f1bc420a118d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzQ4MTE5Yy05YTdjLTRiNWQtOGNkOC1mZWRkZjQ3NjU5ZDUifX0%3D
3002025-03-25T23:00:23.858ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 421 local_addr = [fd00:1122:3344:101::b]:56059 method = GET remote_addr = [fd00:1122:3344:101::d]:46144 req_id = b75cc40e-a7d6-455b-ba1d-50f9d117fd92 response_code = 200 uri = /d5542f9d-430d-4b19-a4dc-31cd924093fb
3012025-03-25T23:00:32.157ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 66 reason = Timeout
3022025-03-25T23:00:32.157ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3032025-03-25T23:00:32.157ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 66 reason = Timeout
3042025-03-25T23:00:32.157ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3052025-03-25T23:00:32.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3062025-03-25T23:00:32.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3072025-03-25T23:00:32.159ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3082025-03-25T23:00:32.160ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3092025-03-25T23:00:32.160ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3102025-03-25T23:00:32.161ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.97361ms iteration = 66 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
3112025-03-25T23:00:32.162ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3122025-03-25T23:00:32.163ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3132025-03-25T23:00:32.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3142025-03-25T23:00:32.164ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3152025-03-25T23:00:32.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3162025-03-25T23:00:32.166ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3172025-03-25T23:00:32.169ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 5d9d6f43-fd71-4256-b06d-1ed4c87e191f, lookup_type: ById(5d9d6f43-fd71-4256-b06d-1ed4c87e191f) } result = Ok(())
3182025-03-25T23:00:32.169ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3192025-03-25T23:00:32.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3202025-03-25T23:00:32.175ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f original_time_created = 2025-03-25 22:49:35.383803 UTC target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f time_created = 2025-03-25 22:49:35.383803 UTC
3212025-03-25T23:00:32.175ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.343343ms iteration = 66 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f","time_created":"2025-03-25T22:49:35.383803Z"}
3222025-03-25T23:00:33.858ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 426 local_addr = [fd00:1122:3344:101::b]:56059 method = GET remote_addr = [fd00:1122:3344:101::d]:46144 req_id = 4fd37e67-139b-45da-b3f8-b0359f9341b1 response_code = 200 uri = /d5542f9d-430d-4b19-a4dc-31cd924093fb
3232025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 67 reason = Timeout
3242025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3252025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 67 reason = Timeout
3262025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 12 reason = Timeout
3272025-03-25T23:00:42.158ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
3282025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 23 reason = Timeout
3292025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3302025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
3312025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3322025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 13 reason = Timeout
3332025-03-25T23:00:42.158ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f
3342025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 18.011\u{b5}s iteration = 13 status = {"enabled":false,"target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f"}
3352025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 12 reason = Timeout
3362025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 23 reason = Timeout
3372025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
3382025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 23 reason = Timeout
3392025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
3402025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 14 reason = Timeout
3412025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 12 reason = Timeout
3422025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3432025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 23 reason = Timeout
3442025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3452025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 23 reason = Timeout
3462025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
3472025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 14 reason = Timeout
3482025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 23 reason = Timeout
3492025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 12 reason = Timeout
3502025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
3512025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
3522025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 12 reason = Timeout
3532025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 25 reason = Timeout
3542025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 12 reason = Timeout
3552025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
3562025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
3572025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 23 reason = Timeout
3582025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
3592025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 12 reason = Timeout
3602025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
3612025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 23 reason = Timeout
3622025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
3632025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3642025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 12 reason = Timeout
3652025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
3662025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3672025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 23 reason = Timeout
3682025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
3692025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 23 reason = Timeout
3702025-03-25T23:00:42.158ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
3712025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3722025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 23 reason = Timeout
3732025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3742025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 12 reason = Timeout
3752025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 12 reason = Timeout
3762025-03-25T23:00:42.158ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:50:42.158529504 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
3772025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
3782025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 23 reason = Timeout
3792025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
3802025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 23 reason = Timeout
3812025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
3822025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
3832025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 23 reason = Timeout
3842025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3852025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 23 reason = Timeout
3862025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
3872025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 23 reason = Timeout
3882025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
3892025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3902025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3912025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
3922025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3932025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
3942025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
3952025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
3962025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3972025-03-25T23:00:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3982025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
3992025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
4002025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4012025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4022025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
4032025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
4042025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
4052025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4062025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
4072025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4082025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4092025-03-25T23:00:42.159ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1f1d5f3e-c8d2-4c62-bebc-2b91d1b5029d.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("25bc856c-a30e-4fd6-94c7-e4c386b38177.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("eb0f1e78-4f86-4f9f-bbd8-3669a0f1cb98.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("eb0f1e78-4f86-4f9f-bbd8-3669a0f1cb98.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1815, tv_nsec: 605074970 } })
4102025-03-25T23:00:42.159ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d421061-bd04-4ffe-bd2d-2457b6327712.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5b0364ab-2148-452d-a2c6-c3386f36f703.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("5b0364ab-2148-452d-a2c6-c3386f36f703.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1815, tv_nsec: 605099320 } })
4112025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
4122025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
4132025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
4142025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
4152025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
4162025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4172025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
4182025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.201637ms iteration = 12 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
4192025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
4202025-03-25T23:00:42.159ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.478153ms iteration = 12 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
4212025-03-25T23:00:42.161ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4222025-03-25T23:00:42.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4232025-03-25T23:00:42.162ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4242025-03-25T23:00:42.162ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4252025-03-25T23:00:42.164ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4262025-03-25T23:00:42.164ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 6.179299ms iteration = 23 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
4272025-03-25T23:00:42.165ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
4282025-03-25T23:00:42.165ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4292025-03-25T23:00:42.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4302025-03-25T23:00:42.165ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
4312025-03-25T23:00:42.165ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
4322025-03-25T23:00:42.166ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
4332025-03-25T23:00:42.166ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
4342025-03-25T23:00:42.166ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
4352025-03-25T23:00:42.166ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 8.271226ms iteration = 12 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
4362025-03-25T23:00:42.167ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
4372025-03-25T23:00:42.167ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
4382025-03-25T23:00:42.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
4392025-03-25T23:00:42.168ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4402025-03-25T23:00:42.169ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1815, tv_nsec: 615761520 } })
4412025-03-25T23:00:42.170ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
4422025-03-25T23:00:42.170ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 12.515662ms iteration = 12 status = {"errors":[],"found_port_ok":[]}
4432025-03-25T23:00:42.172ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1815, tv_nsec: 618140764 } })
4442025-03-25T23:00:42.172ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
4452025-03-25T23:00:42.172ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4462025-03-25T23:00:42.172ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4472025-03-25T23:00:42.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4482025-03-25T23:00:42.173ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4492025-03-25T23:00:42.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4502025-03-25T23:00:42.176ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 18.272442ms iteration = 23 status = {"errors":[],"garbage_collect_requested":[]}
4512025-03-25T23:00:42.177ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4522025-03-25T23:00:42.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4532025-03-25T23:00:42.179ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4542025-03-25T23:00:42.182ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
4552025-03-25T23:00:42.182ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
4562025-03-25T23:00:42.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
4572025-03-25T23:00:42.186ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
4582025-03-25T23:00:42.186ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
4592025-03-25T23:00:42.189ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:50:42.158529504 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
4602025-03-25T23:00:42.189ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 31.079771ms iteration = 12 status = {"expiration":"2025-03-25T22:50:42.158529504Z","pruned":[]}
4612025-03-25T23:00:42.190ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4622025-03-25T23:00:42.190ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4632025-03-25T23:00:42.198ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4642025-03-25T23:00:42.198ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4652025-03-25T23:00:42.200ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 41.818092ms iteration = 23 status = {"errors":[],"requests_created_ok":[]}
4662025-03-25T23:00:42.201ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
4672025-03-25T23:00:42.201ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4682025-03-25T23:00:42.201ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
4692025-03-25T23:00:42.201ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4702025-03-25T23:00:42.201ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4712025-03-25T23:00:42.201ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4722025-03-25T23:00:42.214ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4732025-03-25T23:00:42.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4742025-03-25T23:00:42.216ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
4752025-03-25T23:00:42.216ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
4762025-03-25T23:00:42.216ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 58.265113ms iteration = 12 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
4772025-03-25T23:00:42.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4782025-03-25T23:00:42.216ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4792025-03-25T23:00:42.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4802025-03-25T23:00:42.221ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4812025-03-25T23:00:42.224ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4822025-03-25T23:00:42.227ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4832025-03-25T23:00:42.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4842025-03-25T23:00:42.229ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 5d9d6f43-fd71-4256-b06d-1ed4c87e191f, lookup_type: ById(5d9d6f43-fd71-4256-b06d-1ed4c87e191f) } result = Ok(())
4852025-03-25T23:00:42.229ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4862025-03-25T23:00:42.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4872025-03-25T23:00:42.231ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4882025-03-25T23:00:42.234ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
4892025-03-25T23:00:42.234ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4902025-03-25T23:00:42.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4912025-03-25T23:00:42.244ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4922025-03-25T23:00:42.245ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4932025-03-25T23:00:42.247ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4942025-03-25T23:00:42.248ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 90.381667ms iteration = 23 status = {"errors":[],"finish_invoked_ok":[]}
4952025-03-25T23:00:42.251ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f original_time_created = 2025-03-25 22:49:35.383803 UTC target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f time_created = 2025-03-25 22:49:35.383803 UTC
4962025-03-25T23:00:42.251ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 93.14576ms iteration = 67 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f","time_created":"2025-03-25T22:49:35.383803Z"}
4972025-03-25T23:00:42.251ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
4982025-03-25T23:00:42.251ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4992025-03-25T23:00:42.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
5002025-03-25T23:00:42.253ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5012025-03-25T23:00:42.256ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
5022025-03-25T23:00:42.256ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
5032025-03-25T23:00:42.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
5042025-03-25T23:00:42.268ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5052025-03-25T23:00:42.268ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
5062025-03-25T23:00:42.268ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5072025-03-25T23:00:42.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5082025-03-25T23:00:42.276ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 118.730896ms iteration = 67 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
5092025-03-25T23:00:42.276ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5102025-03-25T23:00:42.277ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5112025-03-25T23:00:42.278ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
5122025-03-25T23:00:42.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
5132025-03-25T23:00:42.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
5142025-03-25T23:00:42.280ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5152025-03-25T23:00:42.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
5162025-03-25T23:00:42.280ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5172025-03-25T23:00:42.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5182025-03-25T23:00:42.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5192025-03-25T23:00:42.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
5202025-03-25T23:00:42.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
5212025-03-25T23:00:42.284ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5222025-03-25T23:00:42.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5232025-03-25T23:00:42.295ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5242025-03-25T23:00:42.297ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5252025-03-25T23:00:42.297ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
5262025-03-25T23:00:42.297ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
5272025-03-25T23:00:42.297ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
5282025-03-25T23:00:42.297ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5292025-03-25T23:00:42.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5302025-03-25T23:00:42.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5312025-03-25T23:00:42.301ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5322025-03-25T23:00:42.301ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5332025-03-25T23:00:42.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5342025-03-25T23:00:42.302ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5352025-03-25T23:00:42.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5362025-03-25T23:00:42.302ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 1 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
5372025-03-25T23:00:42.302ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 144.534938ms iteration = 23 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":1,"total_instances":0,"update_sagas_queued":0}
5382025-03-25T23:00:42.303ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
5392025-03-25T23:00:42.303ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5402025-03-25T23:00:42.303ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3a40a18-6483-4a19-ad0e-97f8552a03c0", "content-length": "26", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} })
5412025-03-25T23:00:42.303ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5422025-03-25T23:00:42.305ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
5432025-03-25T23:00:42.306ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
5442025-03-25T23:00:42.309ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5452025-03-25T23:00:42.314ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
5462025-03-25T23:00:42.314ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5472025-03-25T23:00:42.314ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
5482025-03-25T23:00:42.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5492025-03-25T23:00:42.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
5502025-03-25T23:00:42.327ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
5512025-03-25T23:00:42.328ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
5522025-03-25T23:00:42.328ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5532025-03-25T23:00:42.328ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4ed70b5-e1a1-4c2d-9ffc-f36a5cb705bb", "content-length": "26", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} })
5542025-03-25T23:00:42.328ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5552025-03-25T23:00:42.330ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
5562025-03-25T23:00:42.342ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
5572025-03-25T23:00:42.342ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
5582025-03-25T23:00:42.342ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
5592025-03-25T23:00:42.342ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5602025-03-25T23:00:42.342ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5612025-03-25T23:00:42.398ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 239.691583ms iteration = 25 status = {"physical_disks_added":0}
5622025-03-25T23:00:42.399ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
5632025-03-25T23:00:42.400ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc71d8f2-9887-451e-9440-d56b39f3a325", "content-length": "691", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} })
5642025-03-25T23:00:42.400ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
5652025-03-25T23:00:42.400ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
5662025-03-25T23:00:42.400ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
5672025-03-25T23:00:42.402ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 244.265576ms iteration = 23 status = {}
5682025-03-25T23:00:42.403ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5692025-03-25T23:00:42.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5702025-03-25T23:00:42.404ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5712025-03-25T23:00:42.404ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1815, tv_nsec: 850809898 } })
5722025-03-25T23:00:42.405ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619
5732025-03-25T23:00:42.408ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
5742025-03-25T23:00:42.408ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5752025-03-25T23:00:42.408ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5762025-03-25T23:00:42.412ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5772025-03-25T23:00:42.416ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5782025-03-25T23:00:42.416ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5792025-03-25T23:00:42.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5802025-03-25T23:00:42.445ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 286.895447ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
5812025-03-25T23:00:42.447ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
5822025-03-25T23:00:42.447ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "410a3337-47a6-4b92-83f9-5ae5e59ad639", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
5832025-03-25T23:00:42.447ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
5842025-03-25T23:00:42.448ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5852025-03-25T23:00:42.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5862025-03-25T23:00:42.450ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5872025-03-25T23:00:42.450ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5882025-03-25T23:00:42.450ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
5892025-03-25T23:00:42.450ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5902025-03-25T23:00:42.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5912025-03-25T23:00:42.450ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "7910c4af-61a1-45f9-aa84-fdad709da83c", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
5922025-03-25T23:00:42.450ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
5932025-03-25T23:00:42.451ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5942025-03-25T23:00:42.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5952025-03-25T23:00:42.453ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5962025-03-25T23:00:42.453ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5972025-03-25T23:00:42.453ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5982025-03-25T23:00:42.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5992025-03-25T23:00:42.457ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
6002025-03-25T23:00:42.457ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 39030d55-287b-484a-98ee-50f0b968e055 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
6012025-03-25T23:00:42.457ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6022025-03-25T23:00:42.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6032025-03-25T23:00:42.457ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
6042025-03-25T23:00:42.457ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 299.550832ms iteration = 14 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
6052025-03-25T23:00:42.458ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 9af3e790-030d-4974-9b9e-c35f829f8643, lookup_type: ById(9af3e790-030d-4974-9b9e-c35f829f8643) } result = Ok(())
6062025-03-25T23:00:42.458ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6072025-03-25T23:00:42.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6082025-03-25T23:00:42.458ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 299.89279ms iteration = 23 status = {}
6092025-03-25T23:00:42.460ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 301.825553ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6102025-03-25T23:00:42.460ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 301.796503ms iteration = 23 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
6112025-03-25T23:00:42.460ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 302.080759ms iteration = 23 status = 0
6122025-03-25T23:00:42.460ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
6132025-03-25T23:00:42.460ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 6f07f433-517a-46c5-a5fb-dffdd0b3ba20 dns_zone_name = oxide.test found_so_far = 0 version = 2
6142025-03-25T23:00:42.460ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6152025-03-25T23:00:42.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6162025-03-25T23:00:42.461ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
6172025-03-25T23:00:42.461ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 303.694825ms iteration = 23 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
6182025-03-25T23:00:42.462ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6192025-03-25T23:00:42.463ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6202025-03-25T23:00:42.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6212025-03-25T23:00:42.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6222025-03-25T23:00:42.465ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6232025-03-25T23:00:42.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6242025-03-25T23:00:42.470ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 6f07f433-517a-46c5-a5fb-dffdd0b3ba20 dns_zone_name = oxide.test found_so_far = 1 version = 2
6252025-03-25T23:00:42.470ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
6262025-03-25T23:00:42.470ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:49:43.689588 UTC dns_group = external generation = 2
6272025-03-25T23:00:42.470ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 312.472153ms iteration = 12 status = {"generation":2}
6282025-03-25T23:00:42.475ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
6292025-03-25T23:00:42.476ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
6302025-03-25T23:00:42.476ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "7de3d32b-fee6-48ce-a5fc-ca2b482f81f0", "content-length": "2", "date": "Tue, 25 Mar 2025 23:00:41 GMT"} })
6312025-03-25T23:00:42.478ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6322025-03-25T23:00:42.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6332025-03-25T23:00:42.480ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 321.880936ms iteration = 23 status = null
6342025-03-25T23:00:42.484ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6352025-03-25T23:00:42.484ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
6362025-03-25T23:00:42.484ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 325.790503ms iteration = 23 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
6372025-03-25T23:00:42.492ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 zone_address = fd00:1122:3344:101::2
6382025-03-25T23:00:42.492ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6392025-03-25T23:00:42.492ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "792de3d2-7fea-41e4-805f-9c106ea7baa9", "content-length": "26", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} })
6402025-03-25T23:00:42.492ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6412025-03-25T23:00:42.494ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6422025-03-25T23:00:42.495ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619
6432025-03-25T23:00:42.496ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6442025-03-25T23:00:42.498ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6452025-03-25T23:00:42.498ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6462025-03-25T23:00:42.498ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6472025-03-25T23:00:42.498ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
6482025-03-25T23:00:42.498ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6492025-03-25T23:00:42.498ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6502025-03-25T23:00:42.500ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 39030d55-287b-484a-98ee-50f0b968e055 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
6512025-03-25T23:00:42.500ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
6522025-03-25T23:00:42.500ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:49:42.354168 UTC dns_group = internal generation = 1
6532025-03-25T23:00:42.500ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 342.553962ms iteration = 12 status = {"generation":1}
6542025-03-25T23:00:42.529ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
6552025-03-25T23:00:42.529ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 371.100136ms iteration = 12 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
6562025-03-25T23:00:42.534ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
6572025-03-25T23:00:42.534ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "c12dff62-6177-4b48-a9f2-c01025bc4202", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6582025-03-25T23:00:42.534ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6592025-03-25T23:00:42.535ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:49:38.852077Z, time_modified: 2025-03-25T22:49:38.852077Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 526efdec-95ea-41d0-8852-ffb890a1c1a5, db_silo: Silo { identity: SiloIdentity { id: 526efdec-95ea-41d0-8852-ffb890a1c1a5, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:49:43.626129Z, time_modified: 2025-03-25T22:49:43.626129Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "8a79e1b9407ebe7db0e8d46e699071dda0ee97f80cecb334287dbf0849e0f965" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 526efdec-95ea-41d0-8852-ffb890a1c1a5, db_silo: Silo { identity: SiloIdentity { id: 526efdec-95ea-41d0-8852-ffb890a1c1a5, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:49:43.626129Z, time_modified: 2025-03-25T22:49:43.626129Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "8a79e1b9407ebe7db0e8d46e699071dda0ee97f80cecb334287dbf0849e0f965" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
6602025-03-25T23:00:42.535ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 376.916746ms iteration = 12 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"526efdec-95ea-41d0-8852-ffb890a1c1a5","tls_certs":["8a79e1b9407ebe7db0e8d46e699071dda0ee97f80cecb334287dbf0849e0f965"]}},"default_endpoint":{"silo_id":"526efdec-95ea-41d0-8852-ffb890a1c1a5","tls_certs":["8a79e1b9407ebe7db0e8d46e699071dda0ee97f80cecb334287dbf0849e0f965"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
6612025-03-25T23:00:42.536ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "13c5e5ec-90a1-4427-bf64-6f5d837d729c", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6622025-03-25T23:00:42.536ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6632025-03-25T23:00:42.538ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "c51bb518-17c4-489a-a4f0-93fe49176a54", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6642025-03-25T23:00:42.538ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6652025-03-25T23:00:42.540ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6662025-03-25T23:00:42.540ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 382.270377ms iteration = 14 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
6672025-03-25T23:00:42.577ZINFOnexus (BackgroundTasks): VPC route manager sled 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
6682025-03-25T23:00:42.577ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
6692025-03-25T23:00:42.633ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6702025-03-25T23:00:42.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6712025-03-25T23:00:42.635ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 25f1877d-a731-48fe-8967-16ef2fb00f3b, rack_id: 3ef3cc6b-3ffd-47a3-99ef-c57b25776619, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(dcb3f9bf-5ea7-47e3-b699-6c3a8c7459c5) } rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch_location = Switch0
6722025-03-25T23:00:42.635ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6732025-03-25T23:00:42.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6742025-03-25T23:00:42.646ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
6752025-03-25T23:00:42.646ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "9108b785-67b2-4030-bdb1-1df8b2b141b6", "content-length": "134", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} })
6762025-03-25T23:00:42.646ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
6772025-03-25T23:00:42.646ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = dcb3f9bf-5ea7-47e3-b699-6c3a8c7459c5
6782025-03-25T23:00:42.647ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
6792025-03-25T23:00:42.647ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
6802025-03-25T23:00:42.647ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6812025-03-25T23:00:42.647ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6822025-03-25T23:00:42.647ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6832025-03-25T23:00:42.647ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
6842025-03-25T23:00:42.647ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6852025-03-25T23:00:42.647ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
6862025-03-25T23:00:42.647ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619
6872025-03-25T23:00:42.647ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6882025-03-25T23:00:42.647ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6892025-03-25T23:00:42.650ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
6902025-03-25T23:00:42.650ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "b79b3cc2-b1f1-49fd-8797-93d18cdda15d", "content-length": "2", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} })
6912025-03-25T23:00:42.650ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
6922025-03-25T23:00:42.650ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dbd0e2e-d4f6-4e5c-a7da-3620766100c9", "content-length": "52", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} })
6932025-03-25T23:00:42.650ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch_location = Switch0
6942025-03-25T23:00:42.650ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
6952025-03-25T23:00:42.659ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "d37c6d82-efc2-4b08-8900-5503a0d1eb61", "content-length": "372", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} })
6962025-03-25T23:00:42.659ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6972025-03-25T23:00:42.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6982025-03-25T23:00:42.704ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
6992025-03-25T23:00:42.704ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
7002025-03-25T23:00:42.705ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "7b7712df-5ea0-4a54-b89e-9059e5df8e82", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} })
7012025-03-25T23:00:42.705ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7022025-03-25T23:00:42.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7032025-03-25T23:00:42.708ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7042025-03-25T23:00:42.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7052025-03-25T23:00:42.711ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7062025-03-25T23:00:42.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7072025-03-25T23:00:42.715ZINFOnexus (BackgroundTasks): VPC route manager sled 9af3e790-030d-4974-9b9e-c35f829f8643 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
7082025-03-25T23:00:42.715ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
7092025-03-25T23:00:42.718ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "7ca5e502-29d9-4e41-9f92-f8acb3c3a9b2", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} })
7102025-03-25T23:00:42.719ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 560.862653ms iteration = 23 status = {}
7112025-03-25T23:00:42.760ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "3c59c8e8-9a7e-44f5-b7af-43063e4225fb", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} })
7122025-03-25T23:00:42.760ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch = Switch0
7132025-03-25T23:00:42.760ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
7142025-03-25T23:00:42.764ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "414dc8b4-ec3b-438d-8bd8-f0d257371126", "content-length": "682", "date": "Tue, 25 Mar 2025 23:00:42 GMT"} })
7152025-03-25T23:00:42.764ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7162025-03-25T23:00:42.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7172025-03-25T23:00:42.768ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7182025-03-25T23:00:42.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7192025-03-25T23:00:42.772ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7202025-03-25T23:00:42.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7212025-03-25T23:00:42.773ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7222025-03-25T23:00:42.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7232025-03-25T23:00:42.781ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 622.687107ms iteration = 23 status = {}
7242025-03-25T23:00:42.984ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
7252025-03-25T23:00:42.984ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
7262025-03-25T23:00:43.015ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
7272025-03-25T23:00:43.015ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:42019
7282025-03-25T23:00:43.015ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:42019 req_id = 862d30e0-6ee4-418a-bcab-efb9d422ef2b roles = RoleSet { roles: {} } uri = /metrics/producers
7292025-03-25T23:00:43.016ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:42019 req_id = 862d30e0-6ee4-418a-bcab-efb9d422ef2b resource = Database result = Ok(()) uri = /metrics/producers
7302025-03-25T23:00:43.051ZINFOnexus: assigned collector to new producer collector_id = aa92c5e2-2c4d-4351-b468-f1bc420a118d file = nexus/src/app/oximeter.rs:76 producer_id = d5542f9d-430d-4b19-a4dc-31cd924093fb
7312025-03-25T23:00:43.052ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 36221 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:42019 req_id = 862d30e0-6ee4-418a-bcab-efb9d422ef2b response_code = 201 uri = /metrics/producers
7322025-03-25T23:00:43.052ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "862d30e0-6ee4-418a-bcab-efb9d422ef2b", "content-length": "41", "date": "Tue, 25 Mar 2025 23:00:43 GMT"} })
7332025-03-25T23:00:43.054ZDEBGnexus (producer-registration-task): registered with nexus successfully
7342025-03-25T23:00:43.054ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
7352025-03-25T23:00:43.858ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 315 local_addr = [fd00:1122:3344:101::b]:56059 method = GET remote_addr = [fd00:1122:3344:101::d]:46144 req_id = bf7ea3f7-812b-4e63-813c-8c03fa1c477f response_code = 200 uri = /d5542f9d-430d-4b19-a4dc-31cd924093fb
7362025-03-25T23:00:52.157ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 68 reason = Timeout
7372025-03-25T23:00:52.157ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7382025-03-25T23:00:52.157ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 68 reason = Timeout
7392025-03-25T23:00:52.157ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7402025-03-25T23:00:52.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7412025-03-25T23:00:52.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7422025-03-25T23:00:52.159ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7432025-03-25T23:00:52.162ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7442025-03-25T23:00:52.162ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7452025-03-25T23:00:52.162ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7462025-03-25T23:00:52.164ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7472025-03-25T23:00:52.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7482025-03-25T23:00:52.165ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7492025-03-25T23:00:52.168ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 5d9d6f43-fd71-4256-b06d-1ed4c87e191f, lookup_type: ById(5d9d6f43-fd71-4256-b06d-1ed4c87e191f) } result = Ok(())
7502025-03-25T23:00:52.168ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7512025-03-25T23:00:52.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7522025-03-25T23:00:52.178ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f original_time_created = 2025-03-25 22:49:35.383803 UTC target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f time_created = 2025-03-25 22:49:35.383803 UTC
7532025-03-25T23:00:52.178ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.676986ms iteration = 68 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f","time_created":"2025-03-25T22:49:35.383803Z"}
7542025-03-25T23:00:52.188ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7552025-03-25T23:00:52.188ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7562025-03-25T23:00:52.217ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.996173ms iteration = 68 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7572025-03-25T23:00:53.857ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 288 local_addr = [fd00:1122:3344:101::b]:56059 method = GET remote_addr = [fd00:1122:3344:101::d]:46144 req_id = 051e7e4a-7918-4c7d-8572-c8e807b8112b response_code = 200 uri = /d5542f9d-430d-4b19-a4dc-31cd924093fb
7582025-03-25T23:01:02.158ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
7592025-03-25T23:01:02.158ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7602025-03-25T23:01:02.158ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
7612025-03-25T23:01:02.158ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7622025-03-25T23:01:02.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7632025-03-25T23:01:02.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7642025-03-25T23:01:02.160ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7652025-03-25T23:01:02.160ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7662025-03-25T23:01:02.160ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7672025-03-25T23:01:02.161ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.647442ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7682025-03-25T23:01:02.163ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7692025-03-25T23:01:02.163ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7702025-03-25T23:01:02.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7712025-03-25T23:01:02.165ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7722025-03-25T23:01:02.166ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7732025-03-25T23:01:02.168ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7742025-03-25T23:01:02.170ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 5d9d6f43-fd71-4256-b06d-1ed4c87e191f, lookup_type: ById(5d9d6f43-fd71-4256-b06d-1ed4c87e191f) } result = Ok(())
7752025-03-25T23:01:02.170ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7762025-03-25T23:01:02.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7772025-03-25T23:01:02.180ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f original_time_created = 2025-03-25 22:49:35.383803 UTC target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f time_created = 2025-03-25 22:49:35.383803 UTC
7782025-03-25T23:01:02.180ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.986126ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f","time_created":"2025-03-25T22:49:35.383803Z"}
7792025-03-25T23:01:03.857ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 355 local_addr = [fd00:1122:3344:101::b]:56059 method = GET remote_addr = [fd00:1122:3344:101::d]:46144 req_id = 74bb7333-4236-4f38-ac0f-c998eb5484ea response_code = 200 uri = /d5542f9d-430d-4b19-a4dc-31cd924093fb
7802025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
7812025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7822025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
7832025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
7842025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7852025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
7862025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
7872025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
7882025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
7892025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7902025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
7912025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
7922025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
7932025-03-25T23:01:12.158ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
7942025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
7952025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7962025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7972025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
7982025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 24 reason = Timeout
7992025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
8002025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8012025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
8022025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
8032025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8042025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8052025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
8062025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
8072025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
8082025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
8092025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
8102025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8112025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
8122025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
8132025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
8142025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
8152025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
8162025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
8172025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 24 reason = Timeout
8182025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8192025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8202025-03-25T23:01:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8212025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
8222025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
8232025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
8242025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8252025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8262025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
8272025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
8282025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8292025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
8302025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
8312025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8322025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8332025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
8342025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8352025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
8362025-03-25T23:01:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8372025-03-25T23:01:12.162ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.355309ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
8382025-03-25T23:01:12.164ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
8392025-03-25T23:01:12.164ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
8402025-03-25T23:01:12.164ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
8412025-03-25T23:01:12.164ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1845, tv_nsec: 610142164 } })
8422025-03-25T23:01:12.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
8432025-03-25T23:01:12.164ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
8442025-03-25T23:01:12.166ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8452025-03-25T23:01:12.168ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 10.269811ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
8462025-03-25T23:01:12.169ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
8472025-03-25T23:01:12.169ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8482025-03-25T23:01:12.169ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8492025-03-25T23:01:12.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8502025-03-25T23:01:12.172ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8512025-03-25T23:01:12.172ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8522025-03-25T23:01:12.174ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8532025-03-25T23:01:12.174ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8542025-03-25T23:01:12.174ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
8552025-03-25T23:01:12.174ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8562025-03-25T23:01:12.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8572025-03-25T23:01:12.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8582025-03-25T23:01:12.175ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
8592025-03-25T23:01:12.175ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8602025-03-25T23:01:12.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8612025-03-25T23:01:12.179ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8622025-03-25T23:01:12.180ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1845, tv_nsec: 625987111 } })
8632025-03-25T23:01:12.180ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
8642025-03-25T23:01:12.180ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
8652025-03-25T23:01:12.181ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
8662025-03-25T23:01:12.181ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
8672025-03-25T23:01:12.181ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
8682025-03-25T23:01:12.183ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
8692025-03-25T23:01:12.183ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8702025-03-25T23:01:12.183ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8712025-03-25T23:01:12.185ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8722025-03-25T23:01:12.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8732025-03-25T23:01:12.186ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8742025-03-25T23:01:12.186ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8752025-03-25T23:01:12.188ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8762025-03-25T23:01:12.188ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8772025-03-25T23:01:12.188ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8782025-03-25T23:01:12.188ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8792025-03-25T23:01:12.191ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
8802025-03-25T23:01:12.191ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8812025-03-25T23:01:12.191ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8822025-03-25T23:01:12.193ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 5d9d6f43-fd71-4256-b06d-1ed4c87e191f, lookup_type: ById(5d9d6f43-fd71-4256-b06d-1ed4c87e191f) } result = Ok(())
8832025-03-25T23:01:12.193ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8842025-03-25T23:01:12.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8852025-03-25T23:01:12.216ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f original_time_created = 2025-03-25 22:49:35.383803 UTC target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f time_created = 2025-03-25 22:49:35.383803 UTC
8862025-03-25T23:01:12.216ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 57.894405ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f","time_created":"2025-03-25T22:49:35.383803Z"}
8872025-03-25T23:01:12.222ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
8882025-03-25T23:01:12.222ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
8892025-03-25T23:01:12.222ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "768064b9-26b3-4c91-ae8b-c9c3d429bd63", "content-length": "26", "date": "Tue, 25 Mar 2025 23:01:12 GMT"} })
8902025-03-25T23:01:12.222ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
8912025-03-25T23:01:12.223ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8922025-03-25T23:01:12.225ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
8932025-03-25T23:01:12.225ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
8942025-03-25T23:01:12.225ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8952025-03-25T23:01:12.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8962025-03-25T23:01:12.229ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1845, tv_nsec: 675097108 } })
8972025-03-25T23:01:12.230ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619
8982025-03-25T23:01:12.234ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8992025-03-25T23:01:12.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9002025-03-25T23:01:12.239ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9012025-03-25T23:01:12.239ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9022025-03-25T23:01:12.240ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 81.64563ms iteration = 24 status = {"errors":[],"requests_created_ok":[]}
9032025-03-25T23:01:12.240ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 82.122411ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
9042025-03-25T23:01:12.243ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9052025-03-25T23:01:12.243ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9062025-03-25T23:01:12.243ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9072025-03-25T23:01:12.243ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9082025-03-25T23:01:12.243ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9092025-03-25T23:01:12.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9102025-03-25T23:01:12.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9112025-03-25T23:01:12.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9122025-03-25T23:01:12.244ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9132025-03-25T23:01:12.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9142025-03-25T23:01:12.246ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 87.644265ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
9152025-03-25T23:01:12.247ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9162025-03-25T23:01:12.247ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9172025-03-25T23:01:12.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9182025-03-25T23:01:12.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9192025-03-25T23:01:12.248ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9202025-03-25T23:01:12.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9212025-03-25T23:01:12.249ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
9222025-03-25T23:01:12.249ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
9232025-03-25T23:01:12.249ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 90.750286ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
9242025-03-25T23:01:12.249ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9252025-03-25T23:01:12.250ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9262025-03-25T23:01:12.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9272025-03-25T23:01:12.251ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
9282025-03-25T23:01:12.251ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
9292025-03-25T23:01:12.251ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
9302025-03-25T23:01:12.251ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 93.050328ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
9312025-03-25T23:01:12.251ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a381314-4b62-4a71-8231-a7092159222e", "content-length": "26", "date": "Tue, 25 Mar 2025 23:01:12 GMT"} })
9322025-03-25T23:01:12.251ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
9332025-03-25T23:01:12.253ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9342025-03-25T23:01:12.253ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 9af3e790-030d-4974-9b9e-c35f829f8643, lookup_type: ById(9af3e790-030d-4974-9b9e-c35f829f8643) } result = Ok(())
9352025-03-25T23:01:12.253ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
9362025-03-25T23:01:12.253ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9372025-03-25T23:01:12.253ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9382025-03-25T23:01:12.255ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 96.970456ms iteration = 24 status = 0
9392025-03-25T23:01:12.256ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9402025-03-25T23:01:12.256ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9412025-03-25T23:01:12.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9422025-03-25T23:01:12.278ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9432025-03-25T23:01:12.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9442025-03-25T23:01:12.300ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 141.577441ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
9452025-03-25T23:01:12.300ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
9462025-03-25T23:01:12.300ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 141.613641ms iteration = 24 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
9472025-03-25T23:01:12.306ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 147.928324ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
9482025-03-25T23:01:12.316ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9492025-03-25T23:01:12.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9502025-03-25T23:01:12.320ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 161.959471ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
9512025-03-25T23:01:12.333ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
9522025-03-25T23:01:12.333ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "0555022f-4175-461a-8b61-26114adaad48", "content-length": "2", "date": "Tue, 25 Mar 2025 23:01:12 GMT"} })
9532025-03-25T23:01:12.336ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
9542025-03-25T23:01:12.336ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
9552025-03-25T23:01:12.338ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 179.828243ms iteration = 24 status = null
9562025-03-25T23:01:12.341ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 zone_address = fd00:1122:3344:101::2
9572025-03-25T23:01:12.341ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
9582025-03-25T23:01:12.341ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a17cdc9c-926d-4309-8301-ac703e6f4a2a", "content-length": "26", "date": "Tue, 25 Mar 2025 23:01:12 GMT"} })
9592025-03-25T23:01:12.341ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
9602025-03-25T23:01:12.344ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619
9612025-03-25T23:01:12.349ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 190.342431ms iteration = 26 status = {"physical_disks_added":0}
9622025-03-25T23:01:12.356ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 197.881341ms iteration = 24 status = {}
9632025-03-25T23:01:12.387ZINFOnexus (BackgroundTasks): VPC route manager sled 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
9642025-03-25T23:01:12.387ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
9652025-03-25T23:01:12.387ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c47b6b0-9aea-41aa-9b97-d6c772f61c34", "content-length": "372", "date": "Tue, 25 Mar 2025 23:01:12 GMT"} })
9662025-03-25T23:01:12.388ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9672025-03-25T23:01:12.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9682025-03-25T23:01:12.391ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9692025-03-25T23:01:12.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9702025-03-25T23:01:12.392ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9712025-03-25T23:01:12.394ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
9722025-03-25T23:01:12.394ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9732025-03-25T23:01:12.395ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9742025-03-25T23:01:12.396ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9752025-03-25T23:01:12.398ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
9762025-03-25T23:01:12.398ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9772025-03-25T23:01:12.398ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9782025-03-25T23:01:12.431ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
9792025-03-25T23:01:12.431ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
9802025-03-25T23:01:12.469ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
9812025-03-25T23:01:12.482ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9822025-03-25T23:01:12.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9832025-03-25T23:01:12.485ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 25f1877d-a731-48fe-8967-16ef2fb00f3b, rack_id: 3ef3cc6b-3ffd-47a3-99ef-c57b25776619, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(dcb3f9bf-5ea7-47e3-b699-6c3a8c7459c5) } rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch_location = Switch0
9842025-03-25T23:01:12.485ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9852025-03-25T23:01:12.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9862025-03-25T23:01:12.495ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
9872025-03-25T23:01:12.495ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bdc83c7-d2cf-4fba-b154-88f3a0c7f1f0", "content-length": "134", "date": "Tue, 25 Mar 2025 23:01:12 GMT"} })
9882025-03-25T23:01:12.495ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
9892025-03-25T23:01:12.495ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = dcb3f9bf-5ea7-47e3-b699-6c3a8c7459c5
9902025-03-25T23:01:12.496ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
9912025-03-25T23:01:12.496ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
9922025-03-25T23:01:12.496ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9932025-03-25T23:01:12.496ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9942025-03-25T23:01:12.496ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9952025-03-25T23:01:12.496ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
9962025-03-25T23:01:12.496ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9972025-03-25T23:01:12.496ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
9982025-03-25T23:01:12.496ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619
9992025-03-25T23:01:12.496ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10002025-03-25T23:01:12.496ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10012025-03-25T23:01:12.498ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
10022025-03-25T23:01:12.499ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3b541be-dc02-4a48-986e-071703349fe0", "content-length": "2", "date": "Tue, 25 Mar 2025 23:01:12 GMT"} })
10032025-03-25T23:01:12.499ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
10042025-03-25T23:01:12.499ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "a25fbd68-cf59-4bbd-ae76-9b4bd5e2b89c", "content-length": "52", "date": "Tue, 25 Mar 2025 23:01:12 GMT"} })
10052025-03-25T23:01:12.499ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch_location = Switch0
10062025-03-25T23:01:12.499ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
10072025-03-25T23:01:12.530ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "6ce7cb2a-2867-4c1b-81d2-137a78c7476e", "date": "Tue, 25 Mar 2025 23:01:12 GMT"} })
10082025-03-25T23:01:12.530ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10092025-03-25T23:01:12.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10102025-03-25T23:01:12.531ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8a3ccbc-966e-4155-8a33-131f48aa0f3a", "content-length": "691", "date": "Tue, 25 Mar 2025 23:01:12 GMT"} })
10112025-03-25T23:01:12.531ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
10122025-03-25T23:01:12.531ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
10132025-03-25T23:01:12.531ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
10142025-03-25T23:01:12.532ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 374.205575ms iteration = 24 status = {}
10152025-03-25T23:01:12.532ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10162025-03-25T23:01:12.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10172025-03-25T23:01:12.536ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10182025-03-25T23:01:12.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10192025-03-25T23:01:12.540ZINFOnexus (BackgroundTasks): VPC route manager sled 9af3e790-030d-4974-9b9e-c35f829f8643 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
10202025-03-25T23:01:12.540ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
10212025-03-25T23:01:12.541ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "18332492-6522-457a-9ce7-a15beff8f568", "date": "Tue, 25 Mar 2025 23:01:12 GMT"} })
10222025-03-25T23:01:12.542ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 384.357994ms iteration = 24 status = {}
10232025-03-25T23:01:12.635ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "7e6de3b4-59ec-4733-bee1-bed5b9c8418f", "date": "Tue, 25 Mar 2025 23:01:12 GMT"} })
10242025-03-25T23:01:12.635ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch = Switch0
10252025-03-25T23:01:12.635ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
10262025-03-25T23:01:12.639ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6e019d1-187e-4e68-bccb-2033d5692f2e", "content-length": "682", "date": "Tue, 25 Mar 2025 23:01:12 GMT"} })
10272025-03-25T23:01:12.639ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10282025-03-25T23:01:12.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10292025-03-25T23:01:12.643ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10302025-03-25T23:01:12.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10312025-03-25T23:01:12.646ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10322025-03-25T23:01:12.646ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10332025-03-25T23:01:12.648ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10342025-03-25T23:01:12.648ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10352025-03-25T23:01:12.655ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 497.260909ms iteration = 24 status = {}
10362025-03-25T23:01:13.857ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 316 local_addr = [fd00:1122:3344:101::b]:56059 method = GET remote_addr = [fd00:1122:3344:101::d]:46144 req_id = cad38a62-fe5e-41c3-a92b-c573720e2554 response_code = 200 uri = /d5542f9d-430d-4b19-a4dc-31cd924093fb
10372025-03-25T23:01:22.158ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
10382025-03-25T23:01:22.158ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
10392025-03-25T23:01:22.158ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10402025-03-25T23:01:22.158ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10412025-03-25T23:01:22.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10422025-03-25T23:01:22.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10432025-03-25T23:01:22.160ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10442025-03-25T23:01:22.160ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10452025-03-25T23:01:22.160ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10462025-03-25T23:01:22.162ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10472025-03-25T23:01:22.162ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10482025-03-25T23:01:22.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10492025-03-25T23:01:22.165ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10502025-03-25T23:01:22.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10512025-03-25T23:01:22.166ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10522025-03-25T23:01:22.169ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 5d9d6f43-fd71-4256-b06d-1ed4c87e191f, lookup_type: ById(5d9d6f43-fd71-4256-b06d-1ed4c87e191f) } result = Ok(())
10532025-03-25T23:01:22.169ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10542025-03-25T23:01:22.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10552025-03-25T23:01:22.178ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f original_time_created = 2025-03-25 22:49:35.383803 UTC target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f time_created = 2025-03-25 22:49:35.383803 UTC
10562025-03-25T23:01:22.178ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.027862ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f","time_created":"2025-03-25T22:49:35.383803Z"}
10572025-03-25T23:01:22.190ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.320739ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10582025-03-25T23:01:23.841ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:49912
10592025-03-25T23:01:23.841ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49912 req_id = 23edecf5-63a2-4887-8937-262cd026b423 roles = RoleSet { roles: {} } uri = /metrics/collectors/aa92c5e2-2c4d-4351-b468-f1bc420a118d/producers?sort_by=id_ascending
10602025-03-25T23:01:23.842ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49912 req_id = 23edecf5-63a2-4887-8937-262cd026b423 resource = Database result = Ok(()) uri = /metrics/collectors/aa92c5e2-2c4d-4351-b468-f1bc420a118d/producers?sort_by=id_ascending
10612025-03-25T23:01:23.845ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3658 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49912 req_id = 23edecf5-63a2-4887-8937-262cd026b423 response_code = 200 uri = /metrics/collectors/aa92c5e2-2c4d-4351-b468-f1bc420a118d/producers?sort_by=id_ascending
10622025-03-25T23:01:23.845ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49912 req_id = da155710-a700-4017-8129-f5ac50751023 roles = RoleSet { roles: {} } uri = /metrics/collectors/aa92c5e2-2c4d-4351-b468-f1bc420a118d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzQ4MTE5Yy05YTdjLTRiNWQtOGNkOC1mZWRkZjQ3NjU5ZDUifX0%3D
10632025-03-25T23:01:23.845ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49912 req_id = da155710-a700-4017-8129-f5ac50751023 resource = Database result = Ok(()) uri = /metrics/collectors/aa92c5e2-2c4d-4351-b468-f1bc420a118d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzQ4MTE5Yy05YTdjLTRiNWQtOGNkOC1mZWRkZjQ3NjU5ZDUifX0%3D
10642025-03-25T23:01:23.848ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2501 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49912 req_id = da155710-a700-4017-8129-f5ac50751023 response_code = 200 uri = /metrics/collectors/aa92c5e2-2c4d-4351-b468-f1bc420a118d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzQ4MTE5Yy05YTdjLTRiNWQtOGNkOC1mZWRkZjQ3NjU5ZDUifX0%3D
10652025-03-25T23:01:23.858ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 305 local_addr = [fd00:1122:3344:101::b]:56059 method = GET remote_addr = [fd00:1122:3344:101::d]:46144 req_id = 54cf7db5-a08f-4a31-ad61-29fd223c46cd response_code = 200 uri = /d5542f9d-430d-4b19-a4dc-31cd924093fb
10662025-03-25T23:01:32.158ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 72 reason = Timeout
10672025-03-25T23:01:32.158ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10682025-03-25T23:01:32.158ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 72 reason = Timeout
10692025-03-25T23:01:32.158ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10702025-03-25T23:01:32.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10712025-03-25T23:01:32.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10722025-03-25T23:01:32.160ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10732025-03-25T23:01:32.160ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10742025-03-25T23:01:32.160ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10752025-03-25T23:01:32.162ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.182044ms iteration = 72 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10762025-03-25T23:01:32.163ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10772025-03-25T23:01:32.163ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10782025-03-25T23:01:32.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10792025-03-25T23:01:32.165ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10802025-03-25T23:01:32.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10812025-03-25T23:01:32.167ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10822025-03-25T23:01:32.170ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 5d9d6f43-fd71-4256-b06d-1ed4c87e191f, lookup_type: ById(5d9d6f43-fd71-4256-b06d-1ed4c87e191f) } result = Ok(())
10832025-03-25T23:01:32.170ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10842025-03-25T23:01:32.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10852025-03-25T23:01:32.179ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f original_time_created = 2025-03-25 22:49:35.383803 UTC target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f time_created = 2025-03-25 22:49:35.383803 UTC
10862025-03-25T23:01:32.179ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.85783ms iteration = 72 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f","time_created":"2025-03-25T22:49:35.383803Z"}
10872025-03-25T23:01:33.858ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 370 local_addr = [fd00:1122:3344:101::b]:56059 method = GET remote_addr = [fd00:1122:3344:101::d]:46144 req_id = 2740b3cc-e7d9-4af1-86ff-37fcebc4ea2f response_code = 200 uri = /d5542f9d-430d-4b19-a4dc-31cd924093fb
10882025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 73 reason = Timeout
10892025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10902025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 25 reason = Timeout
10912025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
10922025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 13 reason = Timeout
10932025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 13 reason = Timeout
10942025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
10952025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
10962025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10972025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 25 reason = Timeout
10982025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 25 reason = Timeout
10992025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11002025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11012025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 25 reason = Timeout
11022025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
11032025-03-25T23:01:42.157ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
11042025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 25 reason = Timeout
11052025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
11062025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 25 reason = Timeout
11072025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
11082025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 25 reason = Timeout
11092025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11102025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 25 reason = Timeout
11112025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11122025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 6 reason = Timeout
11132025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11142025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 27 reason = Timeout
11152025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 15 reason = Timeout
11162025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 13 reason = Timeout
11172025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11182025-03-25T23:01:42.158ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:51:42.158017573 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
11192025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
11202025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 14 reason = Timeout
11212025-03-25T23:01:42.158ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f
11222025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 25 reason = Timeout
11232025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 17.101\u{b5}s iteration = 14 status = {"enabled":false,"target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f"}
11242025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
11252025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 25 reason = Timeout
11262025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 25 reason = Timeout
11272025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
11282025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
11292025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 13 reason = Timeout
11302025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 13 reason = Timeout
11312025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
11322025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
11332025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 13 reason = Timeout
11342025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
11352025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 73 reason = Timeout
11362025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11372025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 13 reason = Timeout
11382025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 25 reason = Timeout
11392025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11402025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 13 reason = Timeout
11412025-03-25T23:01:42.158ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
11422025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
11432025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 15 reason = Timeout
11442025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 13 reason = Timeout
11452025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 25 reason = Timeout
11462025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
11472025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 25 reason = Timeout
11482025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
11492025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 13 reason = Timeout
11502025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
11512025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 25 reason = Timeout
11522025-03-25T23:01:42.158ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
11532025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11542025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 25 reason = Timeout
11552025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11562025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11572025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11582025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11592025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11602025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
11612025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
11622025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
11632025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
11642025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11652025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11662025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11672025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
11682025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11692025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
11702025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
11712025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
11722025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
11732025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
11742025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
11752025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
11762025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11772025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11782025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
11792025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11802025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11812025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
11822025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
11832025-03-25T23:01:42.158ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d421061-bd04-4ffe-bd2d-2457b6327712.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5b0364ab-2148-452d-a2c6-c3386f36f703.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("5b0364ab-2148-452d-a2c6-c3386f36f703.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1875, tv_nsec: 604411734 } })
11842025-03-25T23:01:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11852025-03-25T23:01:42.158ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1f1d5f3e-c8d2-4c62-bebc-2b91d1b5029d.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("25bc856c-a30e-4fd6-94c7-e4c386b38177.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("eb0f1e78-4f86-4f9f-bbd8-3669a0f1cb98.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("eb0f1e78-4f86-4f9f-bbd8-3669a0f1cb98.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1875, tv_nsec: 604425035 } })
11862025-03-25T23:01:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11872025-03-25T23:01:42.159ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11882025-03-25T23:01:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11892025-03-25T23:01:42.159ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11902025-03-25T23:01:42.159ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
11912025-03-25T23:01:42.159ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.205247ms iteration = 13 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
11922025-03-25T23:01:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11932025-03-25T23:01:42.159ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11942025-03-25T23:01:42.159ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
11952025-03-25T23:01:42.159ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.509554ms iteration = 13 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
11962025-03-25T23:01:42.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11972025-03-25T23:01:42.161ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11982025-03-25T23:01:42.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11992025-03-25T23:01:42.161ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12002025-03-25T23:01:42.161ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
12012025-03-25T23:01:42.161ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.790865ms iteration = 25 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
12022025-03-25T23:01:42.162ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12032025-03-25T23:01:42.164ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
12042025-03-25T23:01:42.164ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12052025-03-25T23:01:42.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12062025-03-25T23:01:42.168ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1875, tv_nsec: 614416880 } })
12072025-03-25T23:01:42.169ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12082025-03-25T23:01:42.169ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
12092025-03-25T23:01:42.170ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12102025-03-25T23:01:42.170ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 13.052214ms iteration = 25 status = {"errors":[],"finish_invoked_ok":[]}
12112025-03-25T23:01:42.171ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12122025-03-25T23:01:42.171ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1875, tv_nsec: 617482379 } })
12132025-03-25T23:01:42.172ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
12142025-03-25T23:01:42.173ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
12152025-03-25T23:01:42.173ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12162025-03-25T23:01:42.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12172025-03-25T23:01:42.175ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12182025-03-25T23:01:42.176ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:51:42.158017573 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
12192025-03-25T23:01:42.176ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 18.710802ms iteration = 13 status = {"expiration":"2025-03-25T22:51:42.158017573Z","pruned":[]}
12202025-03-25T23:01:42.178ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
12212025-03-25T23:01:42.178ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
12222025-03-25T23:01:42.178ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
12232025-03-25T23:01:42.190ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12242025-03-25T23:01:42.190ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12252025-03-25T23:01:42.194ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
12262025-03-25T23:01:42.194ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
12272025-03-25T23:01:42.194ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12282025-03-25T23:01:42.194ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12292025-03-25T23:01:42.194ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
12302025-03-25T23:01:42.195ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
12312025-03-25T23:01:42.195ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12322025-03-25T23:01:42.197ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
12332025-03-25T23:01:42.197ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12342025-03-25T23:01:42.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12352025-03-25T23:01:42.198ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12362025-03-25T23:01:42.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
12372025-03-25T23:01:42.198ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12382025-03-25T23:01:42.198ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 40.429591ms iteration = 13 status = {"errors":[],"found_port_ok":[]}
12392025-03-25T23:01:42.201ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12402025-03-25T23:01:42.201ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12412025-03-25T23:01:42.201ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12422025-03-25T23:01:42.203ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12432025-03-25T23:01:42.204ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12442025-03-25T23:01:42.207ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
12452025-03-25T23:01:42.207ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12462025-03-25T23:01:42.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12472025-03-25T23:01:42.208ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12482025-03-25T23:01:42.208ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12492025-03-25T23:01:42.209ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12502025-03-25T23:01:42.211ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 54.12791ms iteration = 73 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12512025-03-25T23:01:42.212ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12522025-03-25T23:01:42.215ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12532025-03-25T23:01:42.215ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12542025-03-25T23:01:42.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12552025-03-25T23:01:42.217ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12562025-03-25T23:01:42.218ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12572025-03-25T23:01:42.218ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12582025-03-25T23:01:42.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12592025-03-25T23:01:42.222ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
12602025-03-25T23:01:42.224ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
12612025-03-25T23:01:42.229ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
12622025-03-25T23:01:42.229ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
12632025-03-25T23:01:42.229ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
12642025-03-25T23:01:42.229ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12652025-03-25T23:01:42.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12662025-03-25T23:01:42.234ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
12672025-03-25T23:01:42.234ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
12682025-03-25T23:01:42.234ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 77.052597ms iteration = 13 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
12692025-03-25T23:01:42.235ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12702025-03-25T23:01:42.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12712025-03-25T23:01:42.240ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12722025-03-25T23:01:42.241ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12732025-03-25T23:01:42.241ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12742025-03-25T23:01:42.241ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
12752025-03-25T23:01:42.243ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
12762025-03-25T23:01:42.243ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 85.511248ms iteration = 25 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
12772025-03-25T23:01:42.245ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 87.274437ms iteration = 25 status = {"errors":[],"garbage_collect_requested":[]}
12782025-03-25T23:01:42.245ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-9af3e790-030d-4974-9b9e-c35f829f8643.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1875, tv_nsec: 691005166 } })
12792025-03-25T23:01:42.245ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12802025-03-25T23:01:42.245ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619
12812025-03-25T23:01:42.249ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12822025-03-25T23:01:42.249ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12832025-03-25T23:01:42.249ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12842025-03-25T23:01:42.249ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
12852025-03-25T23:01:42.249ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
12862025-03-25T23:01:42.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12872025-03-25T23:01:42.249ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12882025-03-25T23:01:42.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
12892025-03-25T23:01:42.250ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12902025-03-25T23:01:42.251ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12912025-03-25T23:01:42.251ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
12922025-03-25T23:01:42.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
12932025-03-25T23:01:42.251ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12942025-03-25T23:01:42.252ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12952025-03-25T23:01:42.255ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12962025-03-25T23:01:42.255ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12972025-03-25T23:01:42.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12982025-03-25T23:01:42.257ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
12992025-03-25T23:01:42.257ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13002025-03-25T23:01:42.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13012025-03-25T23:01:42.257ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
13022025-03-25T23:01:42.257ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13032025-03-25T23:01:42.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13042025-03-25T23:01:42.267ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 109.470708ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
13052025-03-25T23:01:42.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13062025-03-25T23:01:42.272ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 114.110282ms iteration = 25 status = {"errors":[],"requests_created_ok":[]}
13072025-03-25T23:01:42.272ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
13082025-03-25T23:01:42.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13092025-03-25T23:01:42.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13102025-03-25T23:01:42.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13112025-03-25T23:01:42.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13122025-03-25T23:01:42.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13132025-03-25T23:01:42.278ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
13142025-03-25T23:01:42.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
13152025-03-25T23:01:42.279ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13162025-03-25T23:01:42.281ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
13172025-03-25T23:01:42.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13182025-03-25T23:01:42.282ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 5d9d6f43-fd71-4256-b06d-1ed4c87e191f, lookup_type: ById(5d9d6f43-fd71-4256-b06d-1ed4c87e191f) } result = Ok(())
13192025-03-25T23:01:42.282ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13202025-03-25T23:01:42.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13212025-03-25T23:01:42.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
13222025-03-25T23:01:42.283ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
13232025-03-25T23:01:42.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
13242025-03-25T23:01:42.283ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13252025-03-25T23:01:42.284ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
13262025-03-25T23:01:42.284ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
13272025-03-25T23:01:42.284ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
13282025-03-25T23:01:42.284ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13292025-03-25T23:01:42.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13302025-03-25T23:01:42.286ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
13312025-03-25T23:01:42.286ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
13322025-03-25T23:01:42.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
13332025-03-25T23:01:42.290ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13342025-03-25T23:01:42.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13352025-03-25T23:01:42.301ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13362025-03-25T23:01:42.303ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 145.368566ms iteration = 6 status = {"nsuccess":5}
13372025-03-25T23:01:42.303ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 145.107401ms iteration = 25 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
13382025-03-25T23:01:42.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
13392025-03-25T23:01:42.304ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
13402025-03-25T23:01:42.304ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
13412025-03-25T23:01:42.304ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
13422025-03-25T23:01:42.304ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 146.031152ms iteration = 13 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
13432025-03-25T23:01:42.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13442025-03-25T23:01:42.307ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13452025-03-25T23:01:42.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13462025-03-25T23:01:42.309ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13472025-03-25T23:01:42.310ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
13482025-03-25T23:01:42.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
13492025-03-25T23:01:42.311ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f original_time_created = 2025-03-25 22:49:35.383803 UTC target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f time_created = 2025-03-25 22:49:35.383803 UTC
13502025-03-25T23:01:42.311ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 153.0489ms iteration = 73 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f","time_created":"2025-03-25T22:49:35.383803Z"}
13512025-03-25T23:01:42.313ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 9af3e790-030d-4974-9b9e-c35f829f8643, lookup_type: ById(9af3e790-030d-4974-9b9e-c35f829f8643) } result = Ok(())
13522025-03-25T23:01:42.313ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13532025-03-25T23:01:42.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13542025-03-25T23:01:42.324ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
13552025-03-25T23:01:42.324ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
13562025-03-25T23:01:42.324ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
13572025-03-25T23:01:42.325ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
13582025-03-25T23:01:42.327ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
13592025-03-25T23:01:42.328ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13602025-03-25T23:01:42.328ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cd5532c-e027-4484-8bf5-b84ffc7ab858", "content-length": "26", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} })
13612025-03-25T23:01:42.328ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
13622025-03-25T23:01:42.329ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13632025-03-25T23:01:42.329ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13642025-03-25T23:01:42.330ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
13652025-03-25T23:01:42.332ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13662025-03-25T23:01:42.332ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 175.091537ms iteration = 25 status = 0
13672025-03-25T23:01:42.335ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
13682025-03-25T23:01:42.335ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13692025-03-25T23:01:42.335ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13702025-03-25T23:01:42.336ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13712025-03-25T23:01:42.339ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
13722025-03-25T23:01:42.339ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13732025-03-25T23:01:42.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13742025-03-25T23:01:42.340ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
13752025-03-25T23:01:42.342ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
13762025-03-25T23:01:42.342ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 6f07f433-517a-46c5-a5fb-dffdd0b3ba20 dns_zone_name = oxide.test found_so_far = 0 version = 2
13772025-03-25T23:01:42.342ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13782025-03-25T23:01:42.343ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13792025-03-25T23:01:42.345ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13802025-03-25T23:01:42.345ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13812025-03-25T23:01:42.346ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 187.894255ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
13822025-03-25T23:01:42.350ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
13832025-03-25T23:01:42.350ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13842025-03-25T23:01:42.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13852025-03-25T23:01:42.350ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
13862025-03-25T23:01:42.350ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13872025-03-25T23:01:42.351ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13882025-03-25T23:01:42.352ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 6f07f433-517a-46c5-a5fb-dffdd0b3ba20 dns_zone_name = oxide.test found_so_far = 1 version = 2
13892025-03-25T23:01:42.352ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
13902025-03-25T23:01:42.352ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:49:43.689588 UTC dns_group = external generation = 2
13912025-03-25T23:01:42.352ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 194.491214ms iteration = 13 status = {"generation":2}
13922025-03-25T23:01:42.353ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
13932025-03-25T23:01:42.353ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 39030d55-287b-484a-98ee-50f0b968e055 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
13942025-03-25T23:01:42.353ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13952025-03-25T23:01:42.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13962025-03-25T23:01:42.355ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13972025-03-25T23:01:42.359ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
13982025-03-25T23:01:42.360ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 202.062265ms iteration = 25 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
13992025-03-25T23:01:42.360ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
14002025-03-25T23:01:42.360ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14012025-03-25T23:01:42.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14022025-03-25T23:01:42.363ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 39030d55-287b-484a-98ee-50f0b968e055 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
14032025-03-25T23:01:42.363ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
14042025-03-25T23:01:42.363ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:49:42.354168 UTC dns_group = internal generation = 1
14052025-03-25T23:01:42.363ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 205.511843ms iteration = 13 status = {"generation":1}
14062025-03-25T23:01:42.373ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
14072025-03-25T23:01:42.373ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
14082025-03-25T23:01:42.373ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 215.386665ms iteration = 25 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
14092025-03-25T23:01:42.393ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:49:38.852077Z, time_modified: 2025-03-25T22:49:38.852077Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 526efdec-95ea-41d0-8852-ffb890a1c1a5, db_silo: Silo { identity: SiloIdentity { id: 526efdec-95ea-41d0-8852-ffb890a1c1a5, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:49:43.626129Z, time_modified: 2025-03-25T22:49:43.626129Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "8a79e1b9407ebe7db0e8d46e699071dda0ee97f80cecb334287dbf0849e0f965" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 526efdec-95ea-41d0-8852-ffb890a1c1a5, db_silo: Silo { identity: SiloIdentity { id: 526efdec-95ea-41d0-8852-ffb890a1c1a5, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:49:43.626129Z, time_modified: 2025-03-25T22:49:43.626129Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "8a79e1b9407ebe7db0e8d46e699071dda0ee97f80cecb334287dbf0849e0f965" }] }) }
14102025-03-25T23:01:42.393ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 234.939596ms iteration = 13 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"526efdec-95ea-41d0-8852-ffb890a1c1a5","tls_certs":["8a79e1b9407ebe7db0e8d46e699071dda0ee97f80cecb334287dbf0849e0f965"]}},"default_endpoint":{"silo_id":"526efdec-95ea-41d0-8852-ffb890a1c1a5","tls_certs":["8a79e1b9407ebe7db0e8d46e699071dda0ee97f80cecb334287dbf0849e0f965"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
14112025-03-25T23:01:42.412ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 zone_address = fd00:1122:3344:101::2
14122025-03-25T23:01:42.412ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14132025-03-25T23:01:42.412ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8938dd56-6a7a-4842-8e52-da639303ee37", "content-length": "26", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} })
14142025-03-25T23:01:42.412ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14152025-03-25T23:01:42.414ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619
14162025-03-25T23:01:42.419ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
14172025-03-25T23:01:42.426ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
14182025-03-25T23:01:42.426ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 268.012231ms iteration = 13 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
14192025-03-25T23:01:42.431ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14202025-03-25T23:01:42.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14212025-03-25T23:01:42.484ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
14222025-03-25T23:01:42.484ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14232025-03-25T23:01:42.485ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "098ba494-90fa-4b45-8e09-85be17bace2b", "content-length": "26", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} })
14242025-03-25T23:01:42.485ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14252025-03-25T23:01:42.486ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
14262025-03-25T23:01:42.493ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 335.604585ms iteration = 27 status = {"physical_disks_added":0}
14272025-03-25T23:01:42.500ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 342.723756ms iteration = 25 status = {}
14282025-03-25T23:01:42.518ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
14292025-03-25T23:01:42.555ZINFOnexus (BackgroundTasks): VPC route manager sled 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
14302025-03-25T23:01:42.555ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
14312025-03-25T23:01:42.557ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14322025-03-25T23:01:42.559ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
14332025-03-25T23:01:42.559ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14342025-03-25T23:01:42.559ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14352025-03-25T23:01:42.561ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14362025-03-25T23:01:42.562ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
14372025-03-25T23:01:42.562ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14382025-03-25T23:01:42.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14392025-03-25T23:01:42.595ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
14402025-03-25T23:01:42.595ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "c69ad80e-da3b-4c7a-a518-fe276f6574a0", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14412025-03-25T23:01:42.595ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14422025-03-25T23:01:42.597ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "b862dee7-3470-4165-b0fd-98867084308d", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14432025-03-25T23:01:42.597ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14442025-03-25T23:01:42.598ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14452025-03-25T23:01:42.598ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 440.320045ms iteration = 15 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
14462025-03-25T23:01:42.604ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
14472025-03-25T23:01:42.604ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a1558ef-6579-4e92-957a-4cdf414e9e91", "content-length": "2", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} })
14482025-03-25T23:01:42.606ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
14492025-03-25T23:01:42.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
14502025-03-25T23:01:42.608ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 450.913175ms iteration = 25 status = null
14512025-03-25T23:01:42.626ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
14522025-03-25T23:01:42.626ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "b529de6d-9d1f-4257-b8d3-30c5095f5197", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14532025-03-25T23:01:42.626ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14542025-03-25T23:01:42.629ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "e44e018e-d5e3-41f0-bd5b-eed0ec652c2f", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14552025-03-25T23:01:42.629ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14562025-03-25T23:01:42.631ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "b3654c6c-6512-4b13-abe1-8b079b42aab3", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14572025-03-25T23:01:42.631ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14582025-03-25T23:01:42.633ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14592025-03-25T23:01:42.633ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 475.401976ms iteration = 15 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
14602025-03-25T23:01:42.671ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "67b14865-7225-4abb-a00a-1cc30b3ee370", "content-length": "372", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} })
14612025-03-25T23:01:42.671ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14622025-03-25T23:01:42.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14632025-03-25T23:01:42.692ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
14642025-03-25T23:01:42.692ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "b02ab1b8-1604-4621-b4d6-be0d167600d0", "content-length": "691", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} })
14652025-03-25T23:01:42.692ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
14662025-03-25T23:01:42.692ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
14672025-03-25T23:01:42.692ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
14682025-03-25T23:01:42.694ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 536.79349ms iteration = 25 status = {}
14692025-03-25T23:01:42.702ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14702025-03-25T23:01:42.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14712025-03-25T23:01:42.704ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 25f1877d-a731-48fe-8967-16ef2fb00f3b, rack_id: 3ef3cc6b-3ffd-47a3-99ef-c57b25776619, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(dcb3f9bf-5ea7-47e3-b699-6c3a8c7459c5) } rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch_location = Switch0
14722025-03-25T23:01:42.704ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14732025-03-25T23:01:42.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14742025-03-25T23:01:42.714ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
14752025-03-25T23:01:42.714ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
14762025-03-25T23:01:42.714ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
14772025-03-25T23:01:42.715ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "7dbe9236-27fe-47bd-8798-af0e5cfd9e2b", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} })
14782025-03-25T23:01:42.715ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14792025-03-25T23:01:42.715ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "35ca5448-f697-4f0d-829f-138ea61ca966", "content-length": "134", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} })
14802025-03-25T23:01:42.715ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
14812025-03-25T23:01:42.715ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = dcb3f9bf-5ea7-47e3-b699-6c3a8c7459c5
14822025-03-25T23:01:42.715ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14832025-03-25T23:01:42.715ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
14842025-03-25T23:01:42.715ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
14852025-03-25T23:01:42.715ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14862025-03-25T23:01:42.715ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14872025-03-25T23:01:42.715ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14882025-03-25T23:01:42.715ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
14892025-03-25T23:01:42.715ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14902025-03-25T23:01:42.715ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
14912025-03-25T23:01:42.716ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619
14922025-03-25T23:01:42.716ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14932025-03-25T23:01:42.716ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14942025-03-25T23:01:42.717ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14952025-03-25T23:01:42.718ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
14962025-03-25T23:01:42.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14972025-03-25T23:01:42.718ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "f44ed2ae-4a45-4801-b9b9-17c7c85d9b90", "content-length": "2", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} })
14982025-03-25T23:01:42.718ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
14992025-03-25T23:01:42.718ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "c486a5ff-6f02-44dc-a36e-ebdaf0f15781", "content-length": "52", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} })
15002025-03-25T23:01:42.718ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch_location = Switch0
15012025-03-25T23:01:42.718ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
15022025-03-25T23:01:42.720ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15032025-03-25T23:01:42.720ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15042025-03-25T23:01:42.724ZINFOnexus (BackgroundTasks): VPC route manager sled 9af3e790-030d-4974-9b9e-c35f829f8643 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
15052025-03-25T23:01:42.724ZDEBGnexus (BackgroundTasks): client request SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
15062025-03-25T23:01:42.820ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "06ea213e-f7e2-4a2f-a783-bc9eeaf1c4ef", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} })
15072025-03-25T23:01:42.820ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 switch = Switch0
15082025-03-25T23:01:42.820ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
15092025-03-25T23:01:42.820ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 3ef3cc6b-3ffd-47a3-99ef-c57b25776619 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "beda5934-a7aa-4683-8453-2b1c678f7c93", "content-length": "682", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} })
15102025-03-25T23:01:42.820ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15112025-03-25T23:01:42.821ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15122025-03-25T23:01:42.821ZDEBGnexus (BackgroundTasks): client response SledAgent = 9af3e790-030d-4974-9b9e-c35f829f8643 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "c1a241f1-4dd0-4d79-b572-2831b1abd104", "date": "Tue, 25 Mar 2025 23:01:42 GMT"} })
15132025-03-25T23:01:42.822ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 664.397637ms iteration = 25 status = {}
15142025-03-25T23:01:42.824ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15152025-03-25T23:01:42.825ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15162025-03-25T23:01:42.828ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15172025-03-25T23:01:42.828ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15182025-03-25T23:01:42.830ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15192025-03-25T23:01:42.831ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15202025-03-25T23:01:42.838ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 679.774633ms iteration = 25 status = {}
15212025-03-25T23:01:43.858ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 333 local_addr = [fd00:1122:3344:101::b]:56059 method = GET remote_addr = [fd00:1122:3344:101::d]:46144 req_id = 25a11548-6d5d-4302-9fdb-c1d75b78f541 response_code = 200 uri = /d5542f9d-430d-4b19-a4dc-31cd924093fb
15222025-03-25T23:01:52.158ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 74 reason = Timeout
15232025-03-25T23:01:52.158ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 74 reason = Timeout
15242025-03-25T23:01:52.158ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15252025-03-25T23:01:52.158ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15262025-03-25T23:01:52.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15272025-03-25T23:01:52.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15282025-03-25T23:01:52.160ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15292025-03-25T23:01:52.160ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15302025-03-25T23:01:52.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15312025-03-25T23:01:52.163ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15322025-03-25T23:01:52.163ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15332025-03-25T23:01:52.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15342025-03-25T23:01:52.165ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15352025-03-25T23:01:52.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15362025-03-25T23:01:52.166ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15372025-03-25T23:01:52.169ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 5d9d6f43-fd71-4256-b06d-1ed4c87e191f, lookup_type: ById(5d9d6f43-fd71-4256-b06d-1ed4c87e191f) } result = Ok(())
15382025-03-25T23:01:52.169ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15392025-03-25T23:01:52.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15402025-03-25T23:01:52.179ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f original_time_created = 2025-03-25 22:49:35.383803 UTC target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f time_created = 2025-03-25 22:49:35.383803 UTC
15412025-03-25T23:01:52.179ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.474231ms iteration = 74 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f","time_created":"2025-03-25T22:49:35.383803Z"}
15422025-03-25T23:01:52.192ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.103229ms iteration = 74 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15432025-03-25T23:01:53.858ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 318 local_addr = [fd00:1122:3344:101::b]:56059 method = GET remote_addr = [fd00:1122:3344:101::d]:46144 req_id = 437f65e1-2cd3-4a44-bdca-b4f1c91d9834 response_code = 200 uri = /d5542f9d-430d-4b19-a4dc-31cd924093fb
15442025-03-25T23:02:02.158ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 75 reason = Timeout
15452025-03-25T23:02:02.158ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15462025-03-25T23:02:02.158ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 75 reason = Timeout
15472025-03-25T23:02:02.158ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15482025-03-25T23:02:02.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15492025-03-25T23:02:02.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15502025-03-25T23:02:02.159ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15512025-03-25T23:02:02.160ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15522025-03-25T23:02:02.160ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15532025-03-25T23:02:02.163ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15542025-03-25T23:02:02.163ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15552025-03-25T23:02:02.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15562025-03-25T23:02:02.165ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15572025-03-25T23:02:02.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15582025-03-25T23:02:02.167ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15592025-03-25T23:02:02.169ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 5d9d6f43-fd71-4256-b06d-1ed4c87e191f, lookup_type: ById(5d9d6f43-fd71-4256-b06d-1ed4c87e191f) } result = Ok(())
15602025-03-25T23:02:02.169ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15612025-03-25T23:02:02.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15622025-03-25T23:02:02.176ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f original_time_created = 2025-03-25 22:49:35.383803 UTC target_id = 5d9d6f43-fd71-4256-b06d-1ed4c87e191f time_created = 2025-03-25 22:49:35.383803 UTC
15632025-03-25T23:02:02.176ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.322923ms iteration = 75 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"5d9d6f43-fd71-4256-b06d-1ed4c87e191f","time_created":"2025-03-25T22:49:35.383803Z"}
15642025-03-25T23:02:02.189ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.468609ms iteration = 75 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15652025-03-25T23:02:03.859ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 355 local_addr = [fd00:1122:3344:101::b]:56059 method = GET remote_addr = [fd00:1122:3344:101::d]:46144 req_id = cc681b89-e5d0-4762-bc70-65b0658e03ae response_code = 200 uri = /d5542f9d-430d-4b19-a4dc-31cd924093fb