Max level shown:
12025-03-25T22:30:03.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 4 reason = Timeout
22025-03-25T22:30:03.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 4 reason = Timeout
32025-03-25T22:30:03.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 4 reason = Timeout
42025-03-25T22:30:03.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
52025-03-25T22:30:03.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
62025-03-25T22:30:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
72025-03-25T22:30:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
82025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 4 reason = Timeout
92025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 4 reason = Timeout
102025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 4 reason = Timeout
112025-03-25T22:30:03.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
122025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
132025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
142025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
152025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 10 reason = Timeout
162025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
172025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
182025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 4 reason = Timeout
192025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 4 reason = Timeout
202025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
212025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
222025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 4 reason = Timeout
232025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 4 reason = Timeout
242025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
252025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
262025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 10 reason = Timeout
272025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 5 reason = Timeout
282025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
292025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
302025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 4 reason = Timeout
312025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 4 reason = Timeout
322025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
332025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
342025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 4 reason = Timeout
352025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
362025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 4 reason = Timeout
372025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
382025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 4 reason = Timeout
392025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 4 reason = Timeout
402025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
412025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
422025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
432025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
442025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
452025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
462025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
472025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
482025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
492025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
502025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
512025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
522025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
532025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
542025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
552025-03-25T22:30:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
562025-03-25T22:30:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
572025-03-25T22:30:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
582025-03-25T22:30:03.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 917, tv_nsec: 365853934 } })
592025-03-25T22:30:03.985ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
602025-03-25T22:30:03.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 917, tv_nsec: 366592737 } })
612025-03-25T22:30:03.986ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
622025-03-25T22:30:03.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
632025-03-25T22:30:03.986ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
642025-03-25T22:30:03.986ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.91779ms iteration = 4 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
652025-03-25T22:30:03.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
662025-03-25T22:30:03.988ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
672025-03-25T22:30:03.988ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
682025-03-25T22:30:03.988ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
692025-03-25T22:30:03.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
702025-03-25T22:30:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
712025-03-25T22:30:03.991ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
722025-03-25T22:30:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
732025-03-25T22:30:03.996ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
742025-03-25T22:30:03.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
752025-03-25T22:30:03.999ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
762025-03-25T22:30:03.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
772025-03-25T22:30:04.003ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
782025-03-25T22:30:04.008ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
792025-03-25T22:30:04.008ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
802025-03-25T22:30:04.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
812025-03-25T22:30:04.011ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
822025-03-25T22:30:04.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
832025-03-25T22:30:04.013ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
842025-03-25T22:30:04.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
852025-03-25T22:30:04.015ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
862025-03-25T22:30:04.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
872025-03-25T22:30:04.019ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 917, tv_nsec: 399831384 } })
882025-03-25T22:30:04.019ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
892025-03-25T22:30:04.038ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 55.657805ms iteration = 4 status = {"errors":[],"garbage_collect_requested":[]}
902025-03-25T22:30:04.046ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
912025-03-25T22:30:04.046ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
922025-03-25T22:30:04.046ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
932025-03-25T22:30:04.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
942025-03-25T22:30:04.052ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
952025-03-25T22:30:04.055ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
962025-03-25T22:30:04.055ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
972025-03-25T22:30:04.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
982025-03-25T22:30:04.055ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
992025-03-25T22:30:04.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1002025-03-25T22:30:04.059ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1012025-03-25T22:30:04.059ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 76.633408ms iteration = 4 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
1022025-03-25T22:30:04.060ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1032025-03-25T22:30:04.061ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 77.748138ms iteration = 4 status = {"errors":[],"finish_invoked_ok":[]}
1042025-03-25T22:30:04.062ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 79.140664ms iteration = 10 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
1052025-03-25T22:30:04.063ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
1062025-03-25T22:30:04.063ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
1072025-03-25T22:30:04.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1082025-03-25T22:30:04.064ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1092025-03-25T22:30:04.064ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1102025-03-25T22:30:04.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1112025-03-25T22:30:04.071ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1122025-03-25T22:30:04.072ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1132025-03-25T22:30:04.075ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1142025-03-25T22:30:04.075ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1152025-03-25T22:30:04.075ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1162025-03-25T22:30:04.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1172025-03-25T22:30:04.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1182025-03-25T22:30:04.077ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1192025-03-25T22:30:04.077ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1202025-03-25T22:30:04.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1212025-03-25T22:30:04.080ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 97.362245ms iteration = 4 status = {"errors":[],"requests_created_ok":[]}
1222025-03-25T22:30:04.082ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1232025-03-25T22:30:04.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1242025-03-25T22:30:04.083ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
1252025-03-25T22:30:04.083ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
1262025-03-25T22:30:04.083ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1272025-03-25T22:30:04.083ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1282025-03-25T22:30:04.083ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "deb1aa96-fa6b-4b88-9107-ef042ceb9233", "content-length": "26", "date": "Tue, 25 Mar 2025 22:30:04 GMT"} })
1292025-03-25T22:30:04.083ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1302025-03-25T22:30:04.084ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1312025-03-25T22:30:04.085ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 102.157344ms iteration = 4 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
1322025-03-25T22:30:04.085ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1332025-03-25T22:30:04.087ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
1342025-03-25T22:30:04.087ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1352025-03-25T22:30:04.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1362025-03-25T22:30:04.092ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
1372025-03-25T22:30:04.092ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1382025-03-25T22:30:04.092ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dd10aaf-7698-4b44-affd-209f5dc54e6d", "content-length": "26", "date": "Tue, 25 Mar 2025 22:30:04 GMT"} })
1392025-03-25T22:30:04.092ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1402025-03-25T22:30:04.093ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1412025-03-25T22:30:04.109ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1422025-03-25T22:30:04.109ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
1432025-03-25T22:30:04.109ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1442025-03-25T22:30:04.110ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1452025-03-25T22:30:04.113ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
1462025-03-25T22:30:04.113ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 130.402348ms iteration = 10 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
1472025-03-25T22:30:04.115ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1482025-03-25T22:30:04.115ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1492025-03-25T22:30:04.117ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1502025-03-25T22:30:04.120ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1512025-03-25T22:30:04.120ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1522025-03-25T22:30:04.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1532025-03-25T22:30:04.131ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
1542025-03-25T22:30:04.131ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 148.841325ms iteration = 4 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
1552025-03-25T22:30:04.132ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1562025-03-25T22:30:04.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1572025-03-25T22:30:04.158ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1582025-03-25T22:30:04.158ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
1592025-03-25T22:30:04.158ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 175.259756ms iteration = 4 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
1602025-03-25T22:30:04.160ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
1612025-03-25T22:30:04.160ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 177.621599ms iteration = 4 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
1622025-03-25T22:30:04.162ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1632025-03-25T22:30:04.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1642025-03-25T22:30:04.166ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1652025-03-25T22:30:04.166ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1662025-03-25T22:30:04.167ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1672025-03-25T22:30:04.171ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
1682025-03-25T22:30:04.171ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1692025-03-25T22:30:04.171ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1702025-03-25T22:30:04.173ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 190.647017ms iteration = 4 status = 0
1712025-03-25T22:30:04.178ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
1722025-03-25T22:30:04.179ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6e4aa04-3ac6-469d-883b-17090d7821b1", "content-length": "1", "date": "Tue, 25 Mar 2025 22:30:04 GMT"} })
1732025-03-25T22:30:04.179ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 195.824541ms iteration = 4 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
1742025-03-25T22:30:04.180ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1752025-03-25T22:30:04.180ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1762025-03-25T22:30:04.181ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
1772025-03-25T22:30:04.182ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1782025-03-25T22:30:04.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1792025-03-25T22:30:04.185ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
1802025-03-25T22:30:04.185ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1812025-03-25T22:30:04.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1822025-03-25T22:30:04.188ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
1832025-03-25T22:30:04.189ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1842025-03-25T22:30:04.189ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "57c7007c-9bef-4486-8993-21a0adc08b3a", "content-length": "26", "date": "Tue, 25 Mar 2025 22:30:04 GMT"} })
1852025-03-25T22:30:04.189ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1862025-03-25T22:30:04.190ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
1872025-03-25T22:30:04.206ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 223.092909ms iteration = 5 status = {"physical_disks_added":0}
1882025-03-25T22:30:04.230ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 248.256728ms iteration = 4 status = {}
1892025-03-25T22:30:04.247ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
1902025-03-25T22:30:04.247ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
1912025-03-25T22:30:04.247ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "872c3e56-cdbc-42b7-b1da-3e7f19272c2e", "content-length": "372", "date": "Tue, 25 Mar 2025 22:30:04 GMT"} })
1922025-03-25T22:30:04.247ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1932025-03-25T22:30:04.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1942025-03-25T22:30:04.249ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1952025-03-25T22:30:04.249ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 267.787624ms iteration = 4 status = null
1962025-03-25T22:30:04.251ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
1972025-03-25T22:30:04.251ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1982025-03-25T22:30:04.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1992025-03-25T22:30:04.297ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
2002025-03-25T22:30:04.298ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed72cfca-79ef-4cf7-8307-f5773ace6926", "content-length": "691", "date": "Tue, 25 Mar 2025 22:30:04 GMT"} })
2012025-03-25T22:30:04.298ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
2022025-03-25T22:30:04.298ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
2032025-03-25T22:30:04.298ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
2042025-03-25T22:30:04.299ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 316.668836ms iteration = 4 status = {}
2052025-03-25T22:30:04.307ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
2062025-03-25T22:30:04.307ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
2072025-03-25T22:30:04.307ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "96365d70-e2af-481f-91a1-65754e13286d", "date": "Tue, 25 Mar 2025 22:30:04 GMT"} })
2082025-03-25T22:30:04.307ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2092025-03-25T22:30:04.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2102025-03-25T22:30:04.310ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2112025-03-25T22:30:04.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2122025-03-25T22:30:04.313ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2132025-03-25T22:30:04.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2142025-03-25T22:30:04.319ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
2152025-03-25T22:30:04.319ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2162025-03-25T22:30:04.319ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "cf700d95-96e4-4100-969e-b01266d15fba", "date": "Tue, 25 Mar 2025 22:30:04 GMT"} })
2172025-03-25T22:30:04.320ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 337.609147ms iteration = 4 status = {}
2182025-03-25T22:30:04.356ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2192025-03-25T22:30:04.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2202025-03-25T22:30:04.360ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
2212025-03-25T22:30:04.360ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2222025-03-25T22:30:04.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2232025-03-25T22:30:04.474ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
2242025-03-25T22:30:04.474ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "79474836-8cbc-42e0-8c8b-18a5499b3638", "content-length": "134", "date": "Tue, 25 Mar 2025 22:30:04 GMT"} })
2252025-03-25T22:30:04.474ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
2262025-03-25T22:30:04.474ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
2272025-03-25T22:30:04.475ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
2282025-03-25T22:30:04.475ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
2292025-03-25T22:30:04.475ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2302025-03-25T22:30:04.475ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2312025-03-25T22:30:04.475ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2322025-03-25T22:30:04.475ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2332025-03-25T22:30:04.475ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2342025-03-25T22:30:04.475ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2352025-03-25T22:30:04.475ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
2362025-03-25T22:30:04.475ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2372025-03-25T22:30:04.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2382025-03-25T22:30:04.484ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
2392025-03-25T22:30:04.484ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "99f67580-a317-4d3b-86cd-0477e3209326", "content-length": "2", "date": "Tue, 25 Mar 2025 22:30:04 GMT"} })
2402025-03-25T22:30:04.484ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
2412025-03-25T22:30:04.484ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0f13013-feb7-4d0b-b38e-880084f1183f", "content-length": "52", "date": "Tue, 25 Mar 2025 22:30:04 GMT"} })
2422025-03-25T22:30:04.484ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
2432025-03-25T22:30:04.484ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
2442025-03-25T22:30:04.623ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "d22a54e2-b668-4a57-af73-cafdf62c80c7", "date": "Tue, 25 Mar 2025 22:30:04 GMT"} })
2452025-03-25T22:30:04.623ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
2462025-03-25T22:30:04.623ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
2472025-03-25T22:30:04.623ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9ba0646-cabf-49af-8bb6-a7a20524fc53", "content-length": "682", "date": "Tue, 25 Mar 2025 22:30:04 GMT"} })
2482025-03-25T22:30:04.623ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2492025-03-25T22:30:04.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2502025-03-25T22:30:04.633ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2512025-03-25T22:30:04.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2522025-03-25T22:30:04.636ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2532025-03-25T22:30:04.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2542025-03-25T22:30:04.747ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2552025-03-25T22:30:04.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2562025-03-25T22:30:04.753ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 770.524713ms iteration = 4 status = {}
2572025-03-25T22:30:06.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 302 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = c82c10c1-cd8d-4f62-be3d-ffb450af909d response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
2582025-03-25T22:30:13.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 11 reason = Timeout
2592025-03-25T22:30:13.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 11 reason = Timeout
2602025-03-25T22:30:13.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2612025-03-25T22:30:13.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2622025-03-25T22:30:13.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2632025-03-25T22:30:13.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2642025-03-25T22:30:13.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2652025-03-25T22:30:13.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2662025-03-25T22:30:13.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2672025-03-25T22:30:13.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.243319ms iteration = 11 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
2682025-03-25T22:30:13.989ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2692025-03-25T22:30:13.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2702025-03-25T22:30:13.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2712025-03-25T22:30:13.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2722025-03-25T22:30:13.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2732025-03-25T22:30:13.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2742025-03-25T22:30:13.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
2752025-03-25T22:30:13.996ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2762025-03-25T22:30:13.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2772025-03-25T22:30:14.002ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
2782025-03-25T22:30:14.002ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.681471ms iteration = 11 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
2792025-03-25T22:30:16.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 289 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = be050701-71e3-45f1-b66b-660466b0f7c1 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
2802025-03-25T22:30:23.982ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 12 reason = Timeout
2812025-03-25T22:30:23.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 12 reason = Timeout
2822025-03-25T22:30:23.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2832025-03-25T22:30:23.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2842025-03-25T22:30:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2852025-03-25T22:30:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2862025-03-25T22:30:23.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2872025-03-25T22:30:23.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2882025-03-25T22:30:23.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2892025-03-25T22:30:23.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.151316ms iteration = 12 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
2902025-03-25T22:30:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2912025-03-25T22:30:23.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2922025-03-25T22:30:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2932025-03-25T22:30:23.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2942025-03-25T22:30:23.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2952025-03-25T22:30:23.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2962025-03-25T22:30:23.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
2972025-03-25T22:30:23.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2982025-03-25T22:30:23.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2992025-03-25T22:30:24.000ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
3002025-03-25T22:30:24.000ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.334606ms iteration = 12 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
3012025-03-25T22:30:26.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 247 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 16fdddda-4ebd-4790-b450-74a8e53f176b response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
3022025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 3 reason = Timeout
3032025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 5 reason = Timeout
3042025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3052025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 5 reason = Timeout
3062025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 3 reason = Timeout
3072025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
3082025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 3 reason = Timeout
3092025-03-25T22:30:33.982ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:20:33.982446726 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
3102025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3112025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
3122025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 3 reason = Timeout
3132025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3142025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 5 reason = Timeout
3152025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 3 reason = Timeout
3162025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
3172025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3182025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 5 reason = Timeout
3192025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 3 reason = Timeout
3202025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
3212025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 5 reason = Timeout
3222025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 5 reason = Timeout
3232025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
3242025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 5 reason = Timeout
3252025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 5 reason = Timeout
3262025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 3 reason = Timeout
3272025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3282025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
3292025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 5 reason = Timeout
3302025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
3312025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 5 reason = Timeout
3322025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3332025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3342025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 4 reason = Timeout
3352025-03-25T22:30:33.982ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b
3362025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 5 reason = Timeout
3372025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 18.83\u{b5}s iteration = 4 status = {"enabled":false,"target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b"}
3382025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
3392025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 5 reason = Timeout
3402025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
3412025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 5 reason = Timeout
3422025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
3432025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 3 reason = Timeout
3442025-03-25T22:30:33.982ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
3452025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
3462025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 5 reason = Timeout
3472025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
3482025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
3492025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 13 reason = Timeout
3502025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3512025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 5 reason = Timeout
3522025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 3 reason = Timeout
3532025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3542025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
3552025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
3562025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 5 reason = Timeout
3572025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 5 reason = Timeout
3582025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
3592025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
3602025-03-25T22:30:33.982ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 13 reason = Timeout
3612025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3622025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 6 reason = Timeout
3632025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3642025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 5 reason = Timeout
3652025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
3662025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 3 reason = Timeout
3672025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
3682025-03-25T22:30:33.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
3692025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3702025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3712025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
3722025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
3732025-03-25T22:30:33.983ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8f1103be-ae21-497e-8fb0-5373f3346d6f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 947, tv_nsec: 364123742 } })
3742025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3752025-03-25T22:30:33.983ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("09fed906-08a5-4e51-9e85-fba7211b9594.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4ba21a7b-c053-4a77-807f-34395131fce2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 947, tv_nsec: 364171433 } })
3762025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
3772025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
3782025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
3792025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3802025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3812025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
3822025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3832025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
3842025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
3852025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
3862025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
3872025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
3882025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
3892025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
3902025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3912025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
3922025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
3932025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
3942025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3952025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3962025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
3972025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.289694ms iteration = 3 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
3982025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
3992025-03-25T22:30:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.451636ms iteration = 3 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
4002025-03-25T22:30:33.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 947, tv_nsec: 366168839 } })
4012025-03-25T22:30:33.985ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4022025-03-25T22:30:33.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4032025-03-25T22:30:33.985ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4042025-03-25T22:30:33.985ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
4052025-03-25T22:30:33.985ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.121947ms iteration = 5 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
4062025-03-25T22:30:33.986ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4072025-03-25T22:30:33.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4082025-03-25T22:30:33.986ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:20:33.982446726 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
4092025-03-25T22:30:33.986ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.138115ms iteration = 3 status = {"expiration":"2025-03-25T22:20:33.982446726Z","pruned":[]}
4102025-03-25T22:30:33.987ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
4112025-03-25T22:30:33.987ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4122025-03-25T22:30:33.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4132025-03-25T22:30:33.989ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4142025-03-25T22:30:33.990ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4152025-03-25T22:30:33.992ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
4162025-03-25T22:30:33.992ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
4172025-03-25T22:30:33.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
4182025-03-25T22:30:33.992ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
4192025-03-25T22:30:33.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
4202025-03-25T22:30:33.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
4212025-03-25T22:30:33.993ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4222025-03-25T22:30:33.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4232025-03-25T22:30:33.994ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4242025-03-25T22:30:33.997ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4252025-03-25T22:30:33.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
4262025-03-25T22:30:33.997ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4272025-03-25T22:30:33.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4282025-03-25T22:30:33.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
4292025-03-25T22:30:33.999ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
4302025-03-25T22:30:33.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
4312025-03-25T22:30:34.006ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4322025-03-25T22:30:34.009ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 947, tv_nsec: 390290189 } })
4332025-03-25T22:30:34.009ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
4342025-03-25T22:30:34.009ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
4352025-03-25T22:30:34.009ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
4362025-03-25T22:30:34.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
4372025-03-25T22:30:34.010ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4382025-03-25T22:30:34.011ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4392025-03-25T22:30:34.012ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
4402025-03-25T22:30:34.012ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4412025-03-25T22:30:34.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4422025-03-25T22:30:34.014ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
4432025-03-25T22:30:34.014ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4442025-03-25T22:30:34.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4452025-03-25T22:30:34.015ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4462025-03-25T22:30:34.015ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4472025-03-25T22:30:34.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4482025-03-25T22:30:34.020ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4492025-03-25T22:30:34.020ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4502025-03-25T22:30:34.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4512025-03-25T22:30:34.021ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4522025-03-25T22:30:34.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4532025-03-25T22:30:34.023ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 947, tv_nsec: 404080261 } })
4542025-03-25T22:30:34.023ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4552025-03-25T22:30:34.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4562025-03-25T22:30:34.023ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
4572025-03-25T22:30:34.028ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4582025-03-25T22:30:34.030ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
4592025-03-25T22:30:34.030ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4602025-03-25T22:30:34.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4612025-03-25T22:30:34.035ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4622025-03-25T22:30:34.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4632025-03-25T22:30:34.045ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
4642025-03-25T22:30:34.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
4652025-03-25T22:30:34.048ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4662025-03-25T22:30:34.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4672025-03-25T22:30:34.063ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
4682025-03-25T22:30:34.063ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
4692025-03-25T22:30:34.063ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
4702025-03-25T22:30:34.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
4712025-03-25T22:30:34.067ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 84.576723ms iteration = 5 status = {"errors":[],"finish_invoked_ok":[]}
4722025-03-25T22:30:34.073ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 91.361097ms iteration = 5 status = {"errors":[],"requests_created_ok":[]}
4732025-03-25T22:30:34.078ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
4742025-03-25T22:30:34.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
4752025-03-25T22:30:34.080ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 97.752773ms iteration = 5 status = {"errors":[],"garbage_collect_requested":[]}
4762025-03-25T22:30:34.080ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
4772025-03-25T22:30:34.080ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
4782025-03-25T22:30:34.080ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 97.855225ms iteration = 3 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
4792025-03-25T22:30:34.082ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4802025-03-25T22:30:34.083ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4812025-03-25T22:30:34.084ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4822025-03-25T22:30:34.087ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
4832025-03-25T22:30:34.087ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4842025-03-25T22:30:34.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4852025-03-25T22:30:34.089ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4862025-03-25T22:30:34.089ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
4872025-03-25T22:30:34.091ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
4882025-03-25T22:30:34.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
4892025-03-25T22:30:34.094ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
4902025-03-25T22:30:34.094ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 111.521074ms iteration = 5 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
4912025-03-25T22:30:34.095ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 112.190177ms iteration = 3 status = {"errors":[],"found_port_ok":[]}
4922025-03-25T22:30:34.095ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4932025-03-25T22:30:34.095ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
4942025-03-25T22:30:34.095ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
4952025-03-25T22:30:34.095ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 112.623694ms iteration = 3 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
4962025-03-25T22:30:34.095ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 113.052751ms iteration = 13 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
4972025-03-25T22:30:34.097ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
4982025-03-25T22:30:34.097ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
4992025-03-25T22:30:34.097ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5002025-03-25T22:30:34.104ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
5012025-03-25T22:30:34.104ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 122.082526ms iteration = 13 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
5022025-03-25T22:30:34.121ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5032025-03-25T22:30:34.122ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5042025-03-25T22:30:34.124ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
5052025-03-25T22:30:34.124ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
5062025-03-25T22:30:34.125ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5072025-03-25T22:30:34.125ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5082025-03-25T22:30:34.127ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5092025-03-25T22:30:34.129ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5102025-03-25T22:30:34.130ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
5112025-03-25T22:30:34.130ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5122025-03-25T22:30:34.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5132025-03-25T22:30:34.132ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
5142025-03-25T22:30:34.133ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5152025-03-25T22:30:34.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5162025-03-25T22:30:34.133ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
5172025-03-25T22:30:34.133ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5182025-03-25T22:30:34.134ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "26a276fa-3414-47bd-9c03-73e9232c2f46", "content-length": "26", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} })
5192025-03-25T22:30:34.134ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5202025-03-25T22:30:34.135ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 152.503841ms iteration = 5 status = 0
5212025-03-25T22:30:34.135ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
5222025-03-25T22:30:34.136ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5232025-03-25T22:30:34.136ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 153.408298ms iteration = 5 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
5242025-03-25T22:30:34.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5252025-03-25T22:30:34.137ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5262025-03-25T22:30:34.139ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
5272025-03-25T22:30:34.139ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5282025-03-25T22:30:34.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5292025-03-25T22:30:34.142ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5302025-03-25T22:30:34.144ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
5312025-03-25T22:30:34.144ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 161.679938ms iteration = 5 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
5322025-03-25T22:30:34.144ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5332025-03-25T22:30:34.144ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
5342025-03-25T22:30:34.144ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5352025-03-25T22:30:34.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5362025-03-25T22:30:34.145ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5372025-03-25T22:30:34.145ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5382025-03-25T22:30:34.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5392025-03-25T22:30:34.151ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
5402025-03-25T22:30:34.151ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
5412025-03-25T22:30:34.151ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
5422025-03-25T22:30:34.151ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5432025-03-25T22:30:34.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5442025-03-25T22:30:34.152ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
5452025-03-25T22:30:34.152ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5462025-03-25T22:30:34.156ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5472025-03-25T22:30:34.157ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5482025-03-25T22:30:34.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5492025-03-25T22:30:34.164ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
5502025-03-25T22:30:34.164ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5512025-03-25T22:30:34.164ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "78990577-1f73-426f-a035-f4dac4376944", "content-length": "26", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} })
5522025-03-25T22:30:34.164ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5532025-03-25T22:30:34.165ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
5542025-03-25T22:30:34.172ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
5552025-03-25T22:30:34.172ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
5562025-03-25T22:30:34.178ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
5572025-03-25T22:30:34.178ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5582025-03-25T22:30:34.178ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e8af5e7-2b95-43bc-ac8e-509206905ff5", "content-length": "26", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} })
5592025-03-25T22:30:34.178ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5602025-03-25T22:30:34.179ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5612025-03-25T22:30:34.179ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
5622025-03-25T22:30:34.179ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 196.384952ms iteration = 5 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
5632025-03-25T22:30:34.180ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
5642025-03-25T22:30:34.185ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 203.113925ms iteration = 5 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
5652025-03-25T22:30:34.189ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
5662025-03-25T22:30:34.189ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 0 version = 2
5672025-03-25T22:30:34.189ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5682025-03-25T22:30:34.189ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5692025-03-25T22:30:34.193ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:28:31.652727Z, time_modified: 2025-03-25T22:28:31.652727Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }) }
5702025-03-25T22:30:34.194ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 211.692111ms iteration = 3 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]}},"default_endpoint":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
5712025-03-25T22:30:34.195ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5722025-03-25T22:30:34.199ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5732025-03-25T22:30:34.199ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5742025-03-25T22:30:34.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5752025-03-25T22:30:34.200ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
5762025-03-25T22:30:34.245ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
5772025-03-25T22:30:34.245ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 262.380246ms iteration = 3 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
5782025-03-25T22:30:34.245ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5792025-03-25T22:30:34.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5802025-03-25T22:30:34.250ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
5812025-03-25T22:30:34.250ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
5822025-03-25T22:30:34.250ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
5832025-03-25T22:30:34.250ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5842025-03-25T22:30:34.250ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5852025-03-25T22:30:34.251ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 268.824233ms iteration = 6 status = {"physical_disks_added":0}
5862025-03-25T22:30:34.277ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
5872025-03-25T22:30:34.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
5882025-03-25T22:30:34.331ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 348.438655ms iteration = 5 status = {}
5892025-03-25T22:30:34.337ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
5902025-03-25T22:30:34.337ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
5912025-03-25T22:30:34.338ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "36b35f54-2313-48c1-a5e3-ba1fbcf1f48c", "content-length": "372", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} })
5922025-03-25T22:30:34.338ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5932025-03-25T22:30:34.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5942025-03-25T22:30:34.340ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 1 version = 2
5952025-03-25T22:30:34.340ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
5962025-03-25T22:30:34.340ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:29:07.710706 UTC dns_group = external generation = 2
5972025-03-25T22:30:34.340ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 358.418287ms iteration = 3 status = {"generation":2}
5982025-03-25T22:30:34.341ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
5992025-03-25T22:30:34.395ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6002025-03-25T22:30:34.398ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
6012025-03-25T22:30:34.398ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6022025-03-25T22:30:34.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6032025-03-25T22:30:34.401ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 418.814149ms iteration = 5 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
6042025-03-25T22:30:34.406ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6052025-03-25T22:30:34.410ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
6062025-03-25T22:30:34.410ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
6072025-03-25T22:30:34.410ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6082025-03-25T22:30:34.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6092025-03-25T22:30:34.410ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e4e7712-33bd-49d4-b1f1-9773c1a89b7a", "content-length": "1", "date": "Tue, 25 Mar 2025 22:30:33 GMT"} })
6102025-03-25T22:30:34.412ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6112025-03-25T22:30:34.412ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6122025-03-25T22:30:34.414ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6132025-03-25T22:30:34.418ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6142025-03-25T22:30:34.418ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6152025-03-25T22:30:34.418ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6162025-03-25T22:30:34.438ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
6172025-03-25T22:30:34.438ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
6182025-03-25T22:30:34.440ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "121746c2-d9d4-4adc-965f-95023263a0d0", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} })
6192025-03-25T22:30:34.440ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6202025-03-25T22:30:34.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6212025-03-25T22:30:34.443ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6222025-03-25T22:30:34.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6232025-03-25T22:30:34.447ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6242025-03-25T22:30:34.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6252025-03-25T22:30:34.452ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
6262025-03-25T22:30:34.452ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
6272025-03-25T22:30:34.452ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "66e30c12-7d0a-4004-b095-fbfb9e812d59", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} })
6282025-03-25T22:30:34.454ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 471.396268ms iteration = 5 status = {}
6292025-03-25T22:30:34.456ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
6302025-03-25T22:30:34.456ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "cb80ea70-9cb1-49cf-af28-9fbec83fe22f", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6312025-03-25T22:30:34.456ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6322025-03-25T22:30:34.458ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "08d6c38f-40fb-45fe-91f7-29941467126d", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6332025-03-25T22:30:34.458ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6342025-03-25T22:30:34.460ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
6352025-03-25T22:30:34.460ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
6362025-03-25T22:30:34.460ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6372025-03-25T22:30:34.460ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6382025-03-25T22:30:34.460ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 478.378335ms iteration = 5 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
6392025-03-25T22:30:34.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6402025-03-25T22:30:34.463ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6412025-03-25T22:30:34.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6422025-03-25T22:30:34.467ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6432025-03-25T22:30:34.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6442025-03-25T22:30:34.468ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
6452025-03-25T22:30:34.468ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "e247824f-339e-478d-9796-e4aad5e0214c", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
6462025-03-25T22:30:34.468ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
6472025-03-25T22:30:34.470ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "25292d25-f8ca-44a3-803d-311bc5aef34e", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
6482025-03-25T22:30:34.470ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
6492025-03-25T22:30:34.472ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
6502025-03-25T22:30:34.472ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "a3240994-b0b6-462b-a17a-1a3797268de8", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
6512025-03-25T22:30:34.472ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
6522025-03-25T22:30:34.472ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
6532025-03-25T22:30:34.472ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:29:06.674111 UTC dns_group = internal generation = 1
6542025-03-25T22:30:34.472ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 490.018237ms iteration = 3 status = {"generation":1}
6552025-03-25T22:30:34.474ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
6562025-03-25T22:30:34.474ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 491.484334ms iteration = 5 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
6572025-03-25T22:30:34.511ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 529.156411ms iteration = 5 status = null
6582025-03-25T22:30:34.540ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6592025-03-25T22:30:34.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6602025-03-25T22:30:34.557ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
6612025-03-25T22:30:34.558ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e96f78da-4b79-4983-9660-c27709e52bb7", "content-length": "691", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} })
6622025-03-25T22:30:34.558ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
6632025-03-25T22:30:34.558ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
6642025-03-25T22:30:34.558ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
6652025-03-25T22:30:34.560ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 577.593744ms iteration = 5 status = {}
6662025-03-25T22:30:34.560ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
6672025-03-25T22:30:34.560ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6682025-03-25T22:30:34.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6692025-03-25T22:30:34.666ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
6702025-03-25T22:30:34.666ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ef3e2ff-aa98-4f65-9440-30febf01abbb", "content-length": "134", "date": "Tue, 25 Mar 2025 22:30:33 GMT"} })
6712025-03-25T22:30:34.666ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
6722025-03-25T22:30:34.666ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
6732025-03-25T22:30:34.666ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
6742025-03-25T22:30:34.666ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
6752025-03-25T22:30:34.666ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6762025-03-25T22:30:34.666ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6772025-03-25T22:30:34.666ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6782025-03-25T22:30:34.666ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
6792025-03-25T22:30:34.667ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6802025-03-25T22:30:34.667ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
6812025-03-25T22:30:34.667ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
6822025-03-25T22:30:34.667ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6832025-03-25T22:30:34.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6842025-03-25T22:30:34.670ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
6852025-03-25T22:30:34.670ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "12744dfb-c40c-4245-a305-b1d18066d938", "content-length": "2", "date": "Tue, 25 Mar 2025 22:30:33 GMT"} })
6862025-03-25T22:30:34.670ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
6872025-03-25T22:30:34.670ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "76ac87fa-0735-4991-88dc-c31be6067bdb", "content-length": "52", "date": "Tue, 25 Mar 2025 22:30:33 GMT"} })
6882025-03-25T22:30:34.670ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
6892025-03-25T22:30:34.670ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
6902025-03-25T22:30:34.775ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "38c9d45e-c177-4851-a6de-f4d26a9afa60", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} })
6912025-03-25T22:30:34.775ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
6922025-03-25T22:30:34.775ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
6932025-03-25T22:30:34.779ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fb85152-9ba6-4b74-ac9c-b1957dd0ae74", "content-length": "682", "date": "Tue, 25 Mar 2025 22:30:34 GMT"} })
6942025-03-25T22:30:34.780ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6952025-03-25T22:30:34.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6962025-03-25T22:30:34.883ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6972025-03-25T22:30:34.884ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6982025-03-25T22:30:34.887ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6992025-03-25T22:30:34.887ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7002025-03-25T22:30:34.889ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7012025-03-25T22:30:34.889ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7022025-03-25T22:30:34.895ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 912.90096ms iteration = 5 status = {}
7032025-03-25T22:30:36.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 446 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = edc9905a-b95f-408c-b67b-046508351abf response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
7042025-03-25T22:30:43.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 14 reason = Timeout
7052025-03-25T22:30:43.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 14 reason = Timeout
7062025-03-25T22:30:43.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7072025-03-25T22:30:43.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7082025-03-25T22:30:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7092025-03-25T22:30:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7102025-03-25T22:30:43.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7112025-03-25T22:30:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7122025-03-25T22:30:43.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7132025-03-25T22:30:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7142025-03-25T22:30:43.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7152025-03-25T22:30:43.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7162025-03-25T22:30:43.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7172025-03-25T22:30:43.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
7182025-03-25T22:30:43.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7192025-03-25T22:30:43.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7202025-03-25T22:30:44.004ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
7212025-03-25T22:30:44.004ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.689436ms iteration = 14 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
7222025-03-25T22:30:44.013ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7232025-03-25T22:30:44.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7242025-03-25T22:30:44.042ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.664478ms iteration = 14 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7252025-03-25T22:30:46.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 291 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 65f6c084-af23-4d3b-99f2-bf941c27178b response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
7262025-03-25T22:30:53.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 15 reason = Timeout
7272025-03-25T22:30:53.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 15 reason = Timeout
7282025-03-25T22:30:53.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7292025-03-25T22:30:53.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7302025-03-25T22:30:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7312025-03-25T22:30:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7322025-03-25T22:30:53.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7332025-03-25T22:30:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7342025-03-25T22:30:53.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7352025-03-25T22:30:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7362025-03-25T22:30:53.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7372025-03-25T22:30:53.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7382025-03-25T22:30:53.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7392025-03-25T22:30:53.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
7402025-03-25T22:30:53.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7412025-03-25T22:30:53.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7422025-03-25T22:30:54.004ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
7432025-03-25T22:30:54.004ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.149526ms iteration = 15 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
7442025-03-25T22:30:54.014ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7452025-03-25T22:30:54.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7462025-03-25T22:30:54.016ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.45712ms iteration = 15 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7472025-03-25T22:30:56.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 291 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = e16f9367-489d-479e-bb55-59504415386a response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
7482025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 6 reason = Timeout
7492025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 6 reason = Timeout
7502025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
7512025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 6 reason = Timeout
7522025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7532025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
7542025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7552025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 16 reason = Timeout
7562025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 6 reason = Timeout
7572025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7582025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 6 reason = Timeout
7592025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
7602025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7612025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 6 reason = Timeout
7622025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
7632025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 6 reason = Timeout
7642025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7652025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 6 reason = Timeout
7662025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
7672025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
7682025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 6 reason = Timeout
7692025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 16 reason = Timeout
7702025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7712025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7722025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 6 reason = Timeout
7732025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
7742025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 6 reason = Timeout
7752025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7762025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 6 reason = Timeout
7772025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 7 reason = Timeout
7782025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7792025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
7802025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 6 reason = Timeout
7812025-03-25T22:31:03.982ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
7822025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7832025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 6 reason = Timeout
7842025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
7852025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 6 reason = Timeout
7862025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
7872025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 6 reason = Timeout
7882025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
7892025-03-25T22:31:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7902025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7912025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7922025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
7932025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
7942025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7952025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
7962025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7972025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7982025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7992025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
8002025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
8012025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8022025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
8032025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8042025-03-25T22:31:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
8052025-03-25T22:31:03.984ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 977, tv_nsec: 365733111 } })
8062025-03-25T22:31:03.985ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
8072025-03-25T22:31:03.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8082025-03-25T22:31:03.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8092025-03-25T22:31:03.985ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.777569ms iteration = 6 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
8102025-03-25T22:31:03.986ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8112025-03-25T22:31:03.987ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8122025-03-25T22:31:03.990ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 977, tv_nsec: 371090709 } })
8132025-03-25T22:31:03.990ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
8142025-03-25T22:31:03.990ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8152025-03-25T22:31:03.990ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
8162025-03-25T22:31:03.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8172025-03-25T22:31:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
8182025-03-25T22:31:03.991ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8192025-03-25T22:31:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8202025-03-25T22:31:03.991ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8212025-03-25T22:31:03.992ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8222025-03-25T22:31:03.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
8232025-03-25T22:31:03.995ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
8242025-03-25T22:31:03.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
8252025-03-25T22:31:03.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
8262025-03-25T22:31:03.996ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8272025-03-25T22:31:03.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8282025-03-25T22:31:03.999ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 16.687704ms iteration = 6 status = {"errors":[],"finish_invoked_ok":[]}
8292025-03-25T22:31:03.999ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8302025-03-25T22:31:04.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8312025-03-25T22:31:04.002ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8322025-03-25T22:31:04.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8332025-03-25T22:31:04.004ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8342025-03-25T22:31:04.006ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8352025-03-25T22:31:04.007ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
8362025-03-25T22:31:04.007ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8372025-03-25T22:31:04.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8382025-03-25T22:31:04.009ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
8392025-03-25T22:31:04.009ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8402025-03-25T22:31:04.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8412025-03-25T22:31:04.011ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 977, tv_nsec: 392763735 } })
8422025-03-25T22:31:04.011ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8432025-03-25T22:31:04.012ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8442025-03-25T22:31:04.012ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
8452025-03-25T22:31:04.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8462025-03-25T22:31:04.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8472025-03-25T22:31:04.012ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8482025-03-25T22:31:04.013ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8492025-03-25T22:31:04.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8502025-03-25T22:31:04.014ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8512025-03-25T22:31:04.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8522025-03-25T22:31:04.015ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
8532025-03-25T22:31:04.015ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8542025-03-25T22:31:04.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8552025-03-25T22:31:04.039ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8562025-03-25T22:31:04.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8572025-03-25T22:31:04.042ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8582025-03-25T22:31:04.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8592025-03-25T22:31:04.045ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8602025-03-25T22:31:04.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8612025-03-25T22:31:04.047ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8622025-03-25T22:31:04.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8632025-03-25T22:31:04.056ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 73.413879ms iteration = 16 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
8642025-03-25T22:31:04.062ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 79.938968ms iteration = 6 status = {"errors":[],"requests_created_ok":[]}
8652025-03-25T22:31:04.062ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
8662025-03-25T22:31:04.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
8672025-03-25T22:31:04.063ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 80.687272ms iteration = 6 status = {"errors":[],"garbage_collect_requested":[]}
8682025-03-25T22:31:04.064ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
8692025-03-25T22:31:04.065ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 82.25691ms iteration = 6 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
8702025-03-25T22:31:04.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
8712025-03-25T22:31:04.066ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
8722025-03-25T22:31:04.066ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 83.586724ms iteration = 6 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
8732025-03-25T22:31:04.068ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8742025-03-25T22:31:04.069ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8752025-03-25T22:31:04.069ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8762025-03-25T22:31:04.069ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
8772025-03-25T22:31:04.069ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
8782025-03-25T22:31:04.069ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
8792025-03-25T22:31:04.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
8802025-03-25T22:31:04.071ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
8812025-03-25T22:31:04.071ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
8822025-03-25T22:31:04.071ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8832025-03-25T22:31:04.071ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
8842025-03-25T22:31:04.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8852025-03-25T22:31:04.071ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bffa1b1b-b447-40f0-8f25-8cdf52f6315a", "content-length": "26", "date": "Tue, 25 Mar 2025 22:31:04 GMT"} })
8862025-03-25T22:31:04.071ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
8872025-03-25T22:31:04.072ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
8882025-03-25T22:31:04.072ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8892025-03-25T22:31:04.073ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
8902025-03-25T22:31:04.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8912025-03-25T22:31:04.073ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
8922025-03-25T22:31:04.073ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8932025-03-25T22:31:04.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8942025-03-25T22:31:04.079ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8952025-03-25T22:31:04.083ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
8962025-03-25T22:31:04.083ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8972025-03-25T22:31:04.083ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8982025-03-25T22:31:04.085ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 102.840355ms iteration = 6 status = 0
8992025-03-25T22:31:04.086ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
9002025-03-25T22:31:04.086ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
9012025-03-25T22:31:04.086ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bd6459b-8355-4c86-bbcb-432811cf9fad", "content-length": "26", "date": "Tue, 25 Mar 2025 22:31:04 GMT"} })
9022025-03-25T22:31:04.086ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
9032025-03-25T22:31:04.087ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
9042025-03-25T22:31:04.093ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
9052025-03-25T22:31:04.093ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 110.691989ms iteration = 16 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
9062025-03-25T22:31:04.102ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
9072025-03-25T22:31:04.102ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
9082025-03-25T22:31:04.102ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9a9c87c-83f0-4f6b-89ec-1752a5c355df", "content-length": "26", "date": "Tue, 25 Mar 2025 22:31:04 GMT"} })
9092025-03-25T22:31:04.102ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
9102025-03-25T22:31:04.103ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
9112025-03-25T22:31:04.104ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 121.959135ms iteration = 6 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
9122025-03-25T22:31:04.110ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 128.168698ms iteration = 6 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
9132025-03-25T22:31:04.112ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
9142025-03-25T22:31:04.112ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 129.880628ms iteration = 6 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
9152025-03-25T22:31:04.120ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
9162025-03-25T22:31:04.120ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
9172025-03-25T22:31:04.120ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
9182025-03-25T22:31:04.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
9192025-03-25T22:31:04.157ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9202025-03-25T22:31:04.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9212025-03-25T22:31:04.160ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
9222025-03-25T22:31:04.160ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
9232025-03-25T22:31:04.160ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 177.955136ms iteration = 6 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
9242025-03-25T22:31:04.174ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 191.953341ms iteration = 6 status = {}
9252025-03-25T22:31:04.176ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
9262025-03-25T22:31:04.178ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ee017a1-3beb-4c29-bfdb-d7a0d5609670", "content-length": "1", "date": "Tue, 25 Mar 2025 22:31:03 GMT"} })
9272025-03-25T22:31:04.180ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
9282025-03-25T22:31:04.181ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
9292025-03-25T22:31:04.203ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 220.695115ms iteration = 7 status = {"physical_disks_added":0}
9302025-03-25T22:31:04.204ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
9312025-03-25T22:31:04.204ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
9322025-03-25T22:31:04.204ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "6664a64c-281c-4a75-858f-31965403d9f4", "content-length": "372", "date": "Tue, 25 Mar 2025 22:31:04 GMT"} })
9332025-03-25T22:31:04.204ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9342025-03-25T22:31:04.205ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9352025-03-25T22:31:04.248ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
9362025-03-25T22:31:04.248ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
9372025-03-25T22:31:04.248ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "bab49277-785b-465c-9961-3cfe95035ed6", "date": "Tue, 25 Mar 2025 22:31:04 GMT"} })
9382025-03-25T22:31:04.248ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9392025-03-25T22:31:04.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9402025-03-25T22:31:04.251ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9412025-03-25T22:31:04.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9422025-03-25T22:31:04.254ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9432025-03-25T22:31:04.254ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9442025-03-25T22:31:04.257ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
9452025-03-25T22:31:04.257ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
9462025-03-25T22:31:04.257ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "427fb13b-3e9d-465b-9916-8e5507044aa1", "date": "Tue, 25 Mar 2025 22:31:04 GMT"} })
9472025-03-25T22:31:04.259ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9482025-03-25T22:31:04.259ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 276.270878ms iteration = 6 status = {}
9492025-03-25T22:31:04.259ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9502025-03-25T22:31:04.261ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
9512025-03-25T22:31:04.261ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9522025-03-25T22:31:04.261ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9532025-03-25T22:31:04.264ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9542025-03-25T22:31:04.264ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9552025-03-25T22:31:04.265ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9562025-03-25T22:31:04.267ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
9572025-03-25T22:31:04.267ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9582025-03-25T22:31:04.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9592025-03-25T22:31:04.268ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9602025-03-25T22:31:04.270ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
9612025-03-25T22:31:04.270ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9622025-03-25T22:31:04.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9632025-03-25T22:31:04.284ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 302.570788ms iteration = 6 status = null
9642025-03-25T22:31:04.308ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
9652025-03-25T22:31:04.387ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
9662025-03-25T22:31:04.388ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c9715d5-34e1-45aa-885a-10bbb65e7d71", "content-length": "134", "date": "Tue, 25 Mar 2025 22:31:04 GMT"} })
9672025-03-25T22:31:04.388ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
9682025-03-25T22:31:04.388ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
9692025-03-25T22:31:04.388ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
9702025-03-25T22:31:04.388ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
9712025-03-25T22:31:04.388ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9722025-03-25T22:31:04.388ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9732025-03-25T22:31:04.388ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9742025-03-25T22:31:04.388ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
9752025-03-25T22:31:04.388ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9762025-03-25T22:31:04.388ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
9772025-03-25T22:31:04.389ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
9782025-03-25T22:31:04.389ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9792025-03-25T22:31:04.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9802025-03-25T22:31:04.391ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
9812025-03-25T22:31:04.391ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "f96cf9f6-775f-473b-b5af-aa9889694598", "content-length": "2", "date": "Tue, 25 Mar 2025 22:31:04 GMT"} })
9822025-03-25T22:31:04.391ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
9832025-03-25T22:31:04.391ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "f68d0ca6-1017-4692-ba77-3c48f739285a", "content-length": "52", "date": "Tue, 25 Mar 2025 22:31:04 GMT"} })
9842025-03-25T22:31:04.391ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
9852025-03-25T22:31:04.391ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
9862025-03-25T22:31:04.439ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "399ba8ff-e90f-4c1d-8a5a-2116921d868b", "content-length": "691", "date": "Tue, 25 Mar 2025 22:31:04 GMT"} })
9872025-03-25T22:31:04.439ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
9882025-03-25T22:31:04.439ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
9892025-03-25T22:31:04.439ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
9902025-03-25T22:31:04.441ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 458.942581ms iteration = 6 status = {}
9912025-03-25T22:31:04.543ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "e2304503-76ad-492f-8cd7-ff4e53b555ac", "date": "Tue, 25 Mar 2025 22:31:04 GMT"} })
9922025-03-25T22:31:04.543ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
9932025-03-25T22:31:04.543ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
9942025-03-25T22:31:04.543ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bbd4c58-edce-4b9c-a7fe-24885cedbbe4", "content-length": "682", "date": "Tue, 25 Mar 2025 22:31:04 GMT"} })
9952025-03-25T22:31:04.543ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9962025-03-25T22:31:04.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9972025-03-25T22:31:04.554ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9982025-03-25T22:31:04.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9992025-03-25T22:31:04.559ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10002025-03-25T22:31:04.559ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10012025-03-25T22:31:04.561ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10022025-03-25T22:31:04.561ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10032025-03-25T22:31:04.575ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 592.691189ms iteration = 6 status = {}
10042025-03-25T22:31:06.083ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 268 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = fd168e0e-c56a-4649-a726-69542b25c71e response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
10052025-03-25T22:31:13.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 17 reason = Timeout
10062025-03-25T22:31:13.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 17 reason = Timeout
10072025-03-25T22:31:13.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10082025-03-25T22:31:13.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10092025-03-25T22:31:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10102025-03-25T22:31:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10112025-03-25T22:31:13.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10122025-03-25T22:31:13.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10132025-03-25T22:31:13.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10142025-03-25T22:31:13.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10152025-03-25T22:31:13.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10162025-03-25T22:31:13.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10172025-03-25T22:31:13.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10182025-03-25T22:31:13.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
10192025-03-25T22:31:13.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10202025-03-25T22:31:13.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10212025-03-25T22:31:14.001ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
10222025-03-25T22:31:14.001ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.277583ms iteration = 17 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
10232025-03-25T22:31:14.022ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10242025-03-25T22:31:14.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10252025-03-25T22:31:14.024ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 41.0846ms iteration = 17 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10262025-03-25T22:31:16.078ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:56582
10272025-03-25T22:31:16.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 8bfe904b-13c7-4895-afa2-3d96a769c2ca roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
10282025-03-25T22:31:16.079ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 8bfe904b-13c7-4895-afa2-3d96a769c2ca resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
10292025-03-25T22:31:16.082ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3361 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 8bfe904b-13c7-4895-afa2-3d96a769c2ca response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
10302025-03-25T22:31:16.082ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = e8d9c5dd-c5d9-41f3-b322-721341d73b44 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
10312025-03-25T22:31:16.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = e8d9c5dd-c5d9-41f3-b322-721341d73b44 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
10322025-03-25T22:31:16.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2146 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = e8d9c5dd-c5d9-41f3-b322-721341d73b44 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
10332025-03-25T22:31:16.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 252 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = abf143de-497a-47ee-8f4a-13a6f0c6d9cc response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
10342025-03-25T22:31:23.982ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 18 reason = Timeout
10352025-03-25T22:31:23.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 18 reason = Timeout
10362025-03-25T22:31:23.982ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10372025-03-25T22:31:23.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10382025-03-25T22:31:23.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10392025-03-25T22:31:23.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10402025-03-25T22:31:23.984ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10412025-03-25T22:31:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10422025-03-25T22:31:23.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10432025-03-25T22:31:23.986ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.648757ms iteration = 18 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10442025-03-25T22:31:23.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10452025-03-25T22:31:23.987ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10462025-03-25T22:31:23.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10472025-03-25T22:31:23.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10482025-03-25T22:31:23.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10492025-03-25T22:31:23.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10502025-03-25T22:31:23.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
10512025-03-25T22:31:23.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10522025-03-25T22:31:23.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10532025-03-25T22:31:23.999ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
10542025-03-25T22:31:23.999ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.915528ms iteration = 18 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
10552025-03-25T22:31:26.083ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 284 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 644346cd-e9a0-41c7-9a8b-c5e862442f6a response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
10562025-03-25T22:31:31.077ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = ecf0f5f5-0a83-4988-996b-7a96095af45e roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
10572025-03-25T22:31:31.078ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = ecf0f5f5-0a83-4988-996b-7a96095af45e resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
10582025-03-25T22:31:31.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3480 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = ecf0f5f5-0a83-4988-996b-7a96095af45e response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
10592025-03-25T22:31:31.081ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 33c73a10-106a-43f6-b9a8-963b113df110 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
10602025-03-25T22:31:31.081ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 33c73a10-106a-43f6-b9a8-963b113df110 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
10612025-03-25T22:31:31.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2582 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 33c73a10-106a-43f6-b9a8-963b113df110 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
10622025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 4 reason = Timeout
10632025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 7 reason = Timeout
10642025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
10652025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 6 reason = Timeout
10662025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 4 reason = Timeout
10672025-03-25T22:31:33.982ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:21:33.982526466 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
10682025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
10692025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 4 reason = Timeout
10702025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10712025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 6 reason = Timeout
10722025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 4 reason = Timeout
10732025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
10742025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 7 reason = Timeout
10752025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10762025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
10772025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 4 reason = Timeout
10782025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 4 reason = Timeout
10792025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
10802025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 7 reason = Timeout
10812025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 4 reason = Timeout
10822025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
10832025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10842025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
10852025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 7 reason = Timeout
10862025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 7 reason = Timeout
10872025-03-25T22:31:33.982ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
10882025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10892025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 7 reason = Timeout
10902025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10912025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10922025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 4 reason = Timeout
10932025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
10942025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 7 reason = Timeout
10952025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
10962025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 7 reason = Timeout
10972025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
10982025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 5 reason = Timeout
10992025-03-25T22:31:33.982ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b
11002025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 19 reason = Timeout
11012025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 15.13\u{b5}s iteration = 5 status = {"enabled":false,"target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b"}
11022025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11032025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 8 reason = Timeout
11042025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 4 reason = Timeout
11052025-03-25T22:31:33.982ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
11062025-03-25T22:31:33.982ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11072025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
11082025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 7 reason = Timeout
11092025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 4 reason = Timeout
11102025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
11112025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11122025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 7 reason = Timeout
11132025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 7 reason = Timeout
11142025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 7 reason = Timeout
11152025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11162025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 7 reason = Timeout
11172025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 7 reason = Timeout
11182025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11192025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
11202025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
11212025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 7 reason = Timeout
11222025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
11232025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
11242025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 7 reason = Timeout
11252025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
11262025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 3 reason = Timeout
11272025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 19 reason = Timeout
11282025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11292025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11302025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11312025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11322025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
11332025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
11342025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11352025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
11362025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
11372025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11382025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
11392025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11402025-03-25T22:31:33.983ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8f1103be-ae21-497e-8fb0-5373f3346d6f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1007, tv_nsec: 364232464 } })
11412025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11422025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
11432025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11442025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
11452025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11462025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
11472025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11482025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
11492025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
11502025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
11512025-03-25T22:31:33.983ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("09fed906-08a5-4e51-9e85-fba7211b9594.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4ba21a7b-c053-4a77-807f-34395131fce2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1007, tv_nsec: 364334146 } })
11522025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
11532025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11542025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11552025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11562025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11572025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
11582025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
11592025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
11602025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11612025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11622025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11632025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11642025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
11652025-03-25T22:31:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11662025-03-25T22:31:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.362565ms iteration = 4 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
11672025-03-25T22:31:33.984ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11682025-03-25T22:31:33.984ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
11692025-03-25T22:31:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.517747ms iteration = 4 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
11702025-03-25T22:31:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11712025-03-25T22:31:33.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1007, tv_nsec: 366357083 } })
11722025-03-25T22:31:33.985ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11732025-03-25T22:31:33.985ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11742025-03-25T22:31:33.985ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
11752025-03-25T22:31:33.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11762025-03-25T22:31:33.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1007, tv_nsec: 367643506 } })
11772025-03-25T22:31:33.987ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:21:33.982526466 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
11782025-03-25T22:31:33.987ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.490772ms iteration = 4 status = {"expiration":"2025-03-25T22:21:33.982526466Z","pruned":[]}
11792025-03-25T22:31:33.987ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
11802025-03-25T22:31:33.987ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
11812025-03-25T22:31:33.987ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11822025-03-25T22:31:33.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11832025-03-25T22:31:33.987ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.726787ms iteration = 7 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
11842025-03-25T22:31:33.988ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
11852025-03-25T22:31:33.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
11862025-03-25T22:31:33.990ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11872025-03-25T22:31:33.990ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
11882025-03-25T22:31:33.990ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
11892025-03-25T22:31:33.990ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 7.799423ms iteration = 4 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
11902025-03-25T22:31:33.990ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 7.876394ms iteration = 4 status = {"errors":[],"found_port_ok":[]}
11912025-03-25T22:31:33.990ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11922025-03-25T22:31:33.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11932025-03-25T22:31:33.993ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11942025-03-25T22:31:33.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11952025-03-25T22:31:33.994ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
11962025-03-25T22:31:33.994ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11972025-03-25T22:31:33.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11982025-03-25T22:31:33.994ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 11.815835ms iteration = 7 status = {"errors":[],"garbage_collect_requested":[]}
11992025-03-25T22:31:33.994ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12002025-03-25T22:31:33.994ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12012025-03-25T22:31:33.994ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 12.113461ms iteration = 7 status = {"errors":[],"requests_created_ok":[]}
12022025-03-25T22:31:33.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
12032025-03-25T22:31:33.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12042025-03-25T22:31:33.998ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12052025-03-25T22:31:34.002ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12062025-03-25T22:31:34.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
12072025-03-25T22:31:34.002ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12082025-03-25T22:31:34.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12092025-03-25T22:31:34.004ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
12102025-03-25T22:31:34.004ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
12112025-03-25T22:31:34.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
12122025-03-25T22:31:34.006ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12132025-03-25T22:31:34.008ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12142025-03-25T22:31:34.010ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
12152025-03-25T22:31:34.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
12162025-03-25T22:31:34.010ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12172025-03-25T22:31:34.010ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12182025-03-25T22:31:34.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12192025-03-25T22:31:34.011ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
12202025-03-25T22:31:34.011ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
12212025-03-25T22:31:34.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
12222025-03-25T22:31:34.012ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12232025-03-25T22:31:34.012ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12242025-03-25T22:31:34.015ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12252025-03-25T22:31:34.015ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12262025-03-25T22:31:34.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12272025-03-25T22:31:34.015ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
12282025-03-25T22:31:34.016ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12292025-03-25T22:31:34.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12302025-03-25T22:31:34.016ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12312025-03-25T22:31:34.019ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12322025-03-25T22:31:34.019ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12332025-03-25T22:31:34.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12342025-03-25T22:31:34.020ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12352025-03-25T22:31:34.022ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12362025-03-25T22:31:34.023ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12372025-03-25T22:31:34.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12382025-03-25T22:31:34.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12392025-03-25T22:31:34.024ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12402025-03-25T22:31:34.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12412025-03-25T22:31:34.025ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12422025-03-25T22:31:34.028ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
12432025-03-25T22:31:34.028ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12442025-03-25T22:31:34.028ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12452025-03-25T22:31:34.029ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12462025-03-25T22:31:34.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12472025-03-25T22:31:34.031ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12482025-03-25T22:31:34.031ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12492025-03-25T22:31:34.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12502025-03-25T22:31:34.035ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 52.203722ms iteration = 19 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12512025-03-25T22:31:34.035ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12522025-03-25T22:31:34.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12532025-03-25T22:31:34.036ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 54.225509ms iteration = 7 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
12542025-03-25T22:31:34.038ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12552025-03-25T22:31:34.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12562025-03-25T22:31:34.044ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12572025-03-25T22:31:34.044ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1007, tv_nsec: 424609316 } })
12582025-03-25T22:31:34.045ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
12592025-03-25T22:31:34.047ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
12602025-03-25T22:31:34.047ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12612025-03-25T22:31:34.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12622025-03-25T22:31:34.055ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12632025-03-25T22:31:34.059ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
12642025-03-25T22:31:34.060ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12652025-03-25T22:31:34.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12662025-03-25T22:31:34.072ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
12672025-03-25T22:31:34.073ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 90.010521ms iteration = 19 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
12682025-03-25T22:31:34.075ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12692025-03-25T22:31:34.076ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
12702025-03-25T22:31:34.076ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
12712025-03-25T22:31:34.076ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12722025-03-25T22:31:34.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
12732025-03-25T22:31:34.077ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12742025-03-25T22:31:34.080ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
12752025-03-25T22:31:34.080ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12762025-03-25T22:31:34.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12772025-03-25T22:31:34.081ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12782025-03-25T22:31:34.081ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12792025-03-25T22:31:34.081ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12802025-03-25T22:31:34.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12812025-03-25T22:31:34.082ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12822025-03-25T22:31:34.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12832025-03-25T22:31:34.083ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 100.569314ms iteration = 7 status = {"errors":[],"finish_invoked_ok":[]}
12842025-03-25T22:31:34.084ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12852025-03-25T22:31:34.084ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
12862025-03-25T22:31:34.084ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
12872025-03-25T22:31:34.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
12882025-03-25T22:31:34.085ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12892025-03-25T22:31:34.085ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
12902025-03-25T22:31:34.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12912025-03-25T22:31:34.086ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12922025-03-25T22:31:34.087ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12932025-03-25T22:31:34.087ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
12942025-03-25T22:31:34.087ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 104.860922ms iteration = 7 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
12952025-03-25T22:31:34.088ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
12962025-03-25T22:31:34.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
12972025-03-25T22:31:34.089ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
12982025-03-25T22:31:34.089ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12992025-03-25T22:31:34.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13002025-03-25T22:31:34.091ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 106.527573ms iteration = 3 status = {"nsuccess":5}
13012025-03-25T22:31:34.091ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13022025-03-25T22:31:34.094ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
13032025-03-25T22:31:34.094ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13042025-03-25T22:31:34.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13052025-03-25T22:31:34.094ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
13062025-03-25T22:31:34.094ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
13072025-03-25T22:31:34.094ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
13082025-03-25T22:31:34.094ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13092025-03-25T22:31:34.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13102025-03-25T22:31:34.096ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13112025-03-25T22:31:34.097ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
13122025-03-25T22:31:34.097ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
13132025-03-25T22:31:34.097ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
13142025-03-25T22:31:34.097ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13152025-03-25T22:31:34.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13162025-03-25T22:31:34.099ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 116.643707ms iteration = 8 status = {"physical_disks_added":0}
13172025-03-25T22:31:34.101ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
13182025-03-25T22:31:34.101ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13192025-03-25T22:31:34.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13202025-03-25T22:31:34.106ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
13212025-03-25T22:31:34.106ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 0 version = 2
13222025-03-25T22:31:34.106ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13232025-03-25T22:31:34.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13242025-03-25T22:31:34.107ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13252025-03-25T22:31:34.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13262025-03-25T22:31:34.110ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13272025-03-25T22:31:34.112ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
13282025-03-25T22:31:34.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
13292025-03-25T22:31:34.115ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13302025-03-25T22:31:34.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13312025-03-25T22:31:34.117ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
13322025-03-25T22:31:34.117ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13332025-03-25T22:31:34.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13342025-03-25T22:31:34.117ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
13352025-03-25T22:31:34.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
13362025-03-25T22:31:34.117ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13372025-03-25T22:31:34.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13382025-03-25T22:31:34.118ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
13392025-03-25T22:31:34.118ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13402025-03-25T22:31:34.118ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba4bbeb4-a4bb-49d1-81d8-c1e73a5e39af", "content-length": "26", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} })
13412025-03-25T22:31:34.119ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
13422025-03-25T22:31:34.120ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
13432025-03-25T22:31:34.120ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13442025-03-25T22:31:34.121ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 1 version = 2
13452025-03-25T22:31:34.121ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
13462025-03-25T22:31:34.121ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:29:07.710706 UTC dns_group = external generation = 2
13472025-03-25T22:31:34.121ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 138.769891ms iteration = 4 status = {"generation":2}
13482025-03-25T22:31:34.121ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13492025-03-25T22:31:34.122ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
13502025-03-25T22:31:34.122ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 139.658117ms iteration = 7 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
13512025-03-25T22:31:34.124ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
13522025-03-25T22:31:34.124ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
13532025-03-25T22:31:34.124ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 141.387038ms iteration = 4 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
13542025-03-25T22:31:34.126ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
13552025-03-25T22:31:34.126ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
13562025-03-25T22:31:34.126ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13572025-03-25T22:31:34.126ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13582025-03-25T22:31:34.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13592025-03-25T22:31:34.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13602025-03-25T22:31:34.128ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 145.88076ms iteration = 7 status = 0
13612025-03-25T22:31:34.131ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
13622025-03-25T22:31:34.145ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
13632025-03-25T22:31:34.145ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 162.602946ms iteration = 7 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
13642025-03-25T22:31:34.147ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
13652025-03-25T22:31:34.157ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 174.151626ms iteration = 7 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
13662025-03-25T22:31:34.166ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 183.975555ms iteration = 7 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
13672025-03-25T22:31:34.170ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13682025-03-25T22:31:34.173ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
13692025-03-25T22:31:34.173ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13702025-03-25T22:31:34.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13712025-03-25T22:31:34.187ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:28:31.652727Z, time_modified: 2025-03-25T22:28:31.652727Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }) }
13722025-03-25T22:31:34.187ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 204.959088ms iteration = 4 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]}},"default_endpoint":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
13732025-03-25T22:31:34.194ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13742025-03-25T22:31:34.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13752025-03-25T22:31:34.203ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
13762025-03-25T22:31:34.203ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 220.795357ms iteration = 4 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
13772025-03-25T22:31:34.235ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
13782025-03-25T22:31:34.235ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13792025-03-25T22:31:34.237ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc16b0f9-871f-4e9f-9e14-fcba4e09690c", "content-length": "26", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} })
13802025-03-25T22:31:34.237ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
13812025-03-25T22:31:34.238ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
13822025-03-25T22:31:34.249ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
13832025-03-25T22:31:34.249ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13842025-03-25T22:31:34.250ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfe2a2c0-4c85-41e0-81c9-11a804fda3b7", "content-length": "26", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} })
13852025-03-25T22:31:34.250ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
13862025-03-25T22:31:34.251ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
13872025-03-25T22:31:34.268ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
13882025-03-25T22:31:34.268ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbd85680-22fe-44cb-b9a4-2125e385bf24", "content-length": "1", "date": "Tue, 25 Mar 2025 22:31:33 GMT"} })
13892025-03-25T22:31:34.269ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
13902025-03-25T22:31:34.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
13912025-03-25T22:31:34.318ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
13922025-03-25T22:31:34.391ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 409.242184ms iteration = 7 status = {}
13932025-03-25T22:31:34.410ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
13942025-03-25T22:31:34.411ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "ff77c70f-2bf9-4b89-99d9-297df0229537", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
13952025-03-25T22:31:34.411ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
13962025-03-25T22:31:34.412ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "aea504a2-8332-4c3f-9d03-afe8b3b91638", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
13972025-03-25T22:31:34.412ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
13982025-03-25T22:31:34.415ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
13992025-03-25T22:31:34.416ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 433.358964ms iteration = 6 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
14002025-03-25T22:31:34.431ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
14012025-03-25T22:31:34.431ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
14022025-03-25T22:31:34.431ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "2441eb78-3572-46bf-80b7-9a8be4678397", "content-length": "372", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} })
14032025-03-25T22:31:34.431ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14042025-03-25T22:31:34.432ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14052025-03-25T22:31:34.435ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 452.766078ms iteration = 7 status = null
14062025-03-25T22:31:34.446ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
14072025-03-25T22:31:34.448ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "0b483d01-e26f-4e3b-96ad-0e025f3e578b", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
14082025-03-25T22:31:34.448ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
14092025-03-25T22:31:34.450ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "eb6637d8-e298-4234-bf8e-ae2f85a2e90c", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
14102025-03-25T22:31:34.450ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
14112025-03-25T22:31:34.452ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "f30b490a-e555-4827-a86d-7b788fa18923", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
14122025-03-25T22:31:34.452ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
14132025-03-25T22:31:34.453ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
14142025-03-25T22:31:34.454ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 471.437848ms iteration = 6 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
14152025-03-25T22:31:34.456ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14162025-03-25T22:31:34.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14172025-03-25T22:31:34.457ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
14182025-03-25T22:31:34.457ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
14192025-03-25T22:31:34.458ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "f460fccf-e8f2-4113-8cc7-45d3c144e752", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} })
14202025-03-25T22:31:34.458ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14212025-03-25T22:31:34.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14222025-03-25T22:31:34.460ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14232025-03-25T22:31:34.461ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
14242025-03-25T22:31:34.462ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14252025-03-25T22:31:34.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14262025-03-25T22:31:34.463ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14272025-03-25T22:31:34.465ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14282025-03-25T22:31:34.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14292025-03-25T22:31:34.467ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
14302025-03-25T22:31:34.467ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14312025-03-25T22:31:34.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14322025-03-25T22:31:34.469ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14332025-03-25T22:31:34.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14342025-03-25T22:31:34.475ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
14352025-03-25T22:31:34.475ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
14362025-03-25T22:31:34.476ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "fd3889d1-aa73-4608-b125-a5b757aec0e2", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} })
14372025-03-25T22:31:34.478ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 496.009786ms iteration = 7 status = {}
14382025-03-25T22:31:34.489ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
14392025-03-25T22:31:34.489ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
14402025-03-25T22:31:34.489ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14412025-03-25T22:31:34.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14422025-03-25T22:31:34.491ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14432025-03-25T22:31:34.495ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
14442025-03-25T22:31:34.495ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14452025-03-25T22:31:34.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14462025-03-25T22:31:34.501ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
14472025-03-25T22:31:34.501ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
14482025-03-25T22:31:34.501ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:29:06.674111 UTC dns_group = internal generation = 1
14492025-03-25T22:31:34.501ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 518.963885ms iteration = 4 status = {"generation":1}
14502025-03-25T22:31:34.542ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14512025-03-25T22:31:34.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14522025-03-25T22:31:34.545ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
14532025-03-25T22:31:34.545ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14542025-03-25T22:31:34.545ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14552025-03-25T22:31:34.557ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
14562025-03-25T22:31:34.558ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc4f9db2-3da0-47c6-8951-966fe4b9fa3f", "content-length": "134", "date": "Tue, 25 Mar 2025 22:31:33 GMT"} })
14572025-03-25T22:31:34.558ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
14582025-03-25T22:31:34.558ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
14592025-03-25T22:31:34.558ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
14602025-03-25T22:31:34.558ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
14612025-03-25T22:31:34.558ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14622025-03-25T22:31:34.558ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14632025-03-25T22:31:34.558ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14642025-03-25T22:31:34.558ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
14652025-03-25T22:31:34.559ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14662025-03-25T22:31:34.559ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
14672025-03-25T22:31:34.559ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
14682025-03-25T22:31:34.559ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14692025-03-25T22:31:34.559ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14702025-03-25T22:31:34.561ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
14712025-03-25T22:31:34.561ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "833dc3f0-32da-4df7-a18a-8e99f86cff28", "content-length": "2", "date": "Tue, 25 Mar 2025 22:31:33 GMT"} })
14722025-03-25T22:31:34.561ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
14732025-03-25T22:31:34.561ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "683ed4ca-5be4-4007-a777-969242f3afd2", "content-length": "52", "date": "Tue, 25 Mar 2025 22:31:33 GMT"} })
14742025-03-25T22:31:34.561ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
14752025-03-25T22:31:34.561ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
14762025-03-25T22:31:34.590ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
14772025-03-25T22:31:34.672ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "794e57fa-ee1e-4e62-a4c4-418d8e11cc15", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} })
14782025-03-25T22:31:34.672ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
14792025-03-25T22:31:34.672ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
14802025-03-25T22:31:34.672ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef254152-d1eb-44c6-b2bf-5788f798a33e", "content-length": "682", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} })
14812025-03-25T22:31:34.672ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14822025-03-25T22:31:34.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14832025-03-25T22:31:34.676ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14842025-03-25T22:31:34.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14852025-03-25T22:31:34.677ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f50a493-573c-48f3-80f3-013d2963847e", "content-length": "691", "date": "Tue, 25 Mar 2025 22:31:34 GMT"} })
14862025-03-25T22:31:34.677ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
14872025-03-25T22:31:34.677ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
14882025-03-25T22:31:34.677ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
14892025-03-25T22:31:34.678ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 695.78117ms iteration = 7 status = {}
14902025-03-25T22:31:34.678ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14912025-03-25T22:31:34.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14922025-03-25T22:31:34.680ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14932025-03-25T22:31:34.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14942025-03-25T22:31:34.687ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 704.961878ms iteration = 7 status = {}
14952025-03-25T22:31:36.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 304 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = e5671e79-9494-4bc8-8cc0-09b9b7ddfa1e response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
14962025-03-25T22:31:43.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 20 reason = Timeout
14972025-03-25T22:31:43.982ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 20 reason = Timeout
14982025-03-25T22:31:43.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14992025-03-25T22:31:43.982ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15002025-03-25T22:31:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15012025-03-25T22:31:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15022025-03-25T22:31:43.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15032025-03-25T22:31:43.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15042025-03-25T22:31:43.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15052025-03-25T22:31:43.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.262528ms iteration = 20 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15062025-03-25T22:31:43.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15072025-03-25T22:31:43.987ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15082025-03-25T22:31:43.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15092025-03-25T22:31:43.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15102025-03-25T22:31:43.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15112025-03-25T22:31:43.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15122025-03-25T22:31:43.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
15132025-03-25T22:31:43.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15142025-03-25T22:31:43.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15152025-03-25T22:31:44.000ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
15162025-03-25T22:31:44.000ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.369377ms iteration = 20 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
15172025-03-25T22:31:46.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 278 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = f77be743-5d7a-4e1a-a2fb-2b427346d48e response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
15182025-03-25T22:31:53.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 21 reason = Timeout
15192025-03-25T22:31:53.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 21 reason = Timeout
15202025-03-25T22:31:53.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15212025-03-25T22:31:53.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15222025-03-25T22:31:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15232025-03-25T22:31:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15242025-03-25T22:31:53.984ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15252025-03-25T22:31:53.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15262025-03-25T22:31:53.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15272025-03-25T22:31:53.986ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.523304ms iteration = 21 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15282025-03-25T22:31:53.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15292025-03-25T22:31:53.987ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15302025-03-25T22:31:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15312025-03-25T22:31:53.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15322025-03-25T22:31:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15332025-03-25T22:31:53.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15342025-03-25T22:31:53.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
15352025-03-25T22:31:53.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15362025-03-25T22:31:53.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15372025-03-25T22:31:54.002ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
15382025-03-25T22:31:54.002ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.20757ms iteration = 21 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
15392025-03-25T22:31:56.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 305 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = e96a43f8-d39c-4bfd-a22d-9a4bcf243fae response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
15402025-03-25T22:32:01.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = feda0fa7-ab79-41ee-abeb-eb0d23a82186 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
15412025-03-25T22:32:01.079ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = feda0fa7-ab79-41ee-abeb-eb0d23a82186 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
15422025-03-25T22:32:01.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2473 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = feda0fa7-ab79-41ee-abeb-eb0d23a82186 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
15432025-03-25T22:32:01.081ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 26075b10-ddba-458a-b01f-0ed50d2e2ee5 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
15442025-03-25T22:32:01.081ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 26075b10-ddba-458a-b01f-0ed50d2e2ee5 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
15452025-03-25T22:32:01.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3074 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 26075b10-ddba-458a-b01f-0ed50d2e2ee5 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
15462025-03-25T22:32:03.981ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 8 reason = Timeout
15472025-03-25T22:32:03.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 8 reason = Timeout
15482025-03-25T22:32:03.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
15492025-03-25T22:32:03.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 8 reason = Timeout
15502025-03-25T22:32:03.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
15512025-03-25T22:32:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
15522025-03-25T22:32:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
15532025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 8 reason = Timeout
15542025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 22 reason = Timeout
15552025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15562025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
15572025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 8 reason = Timeout
15582025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 8 reason = Timeout
15592025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
15602025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15612025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 8 reason = Timeout
15622025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 8 reason = Timeout
15632025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
15642025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
15652025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 8 reason = Timeout
15662025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
15672025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 8 reason = Timeout
15682025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
15692025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 8 reason = Timeout
15702025-03-25T22:32:03.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
15712025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15722025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 8 reason = Timeout
15732025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 8 reason = Timeout
15742025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 22 reason = Timeout
15752025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15762025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
15772025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15782025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 9 reason = Timeout
15792025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15802025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 8 reason = Timeout
15812025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15822025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 8 reason = Timeout
15832025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
15842025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 8 reason = Timeout
15852025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
15862025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
15872025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15882025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15892025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
15902025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15912025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
15922025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
15932025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
15942025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15952025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15962025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
15972025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
15982025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15992025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16002025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16012025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
16022025-03-25T22:32:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
16032025-03-25T22:32:03.991ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1037, tv_nsec: 372063747 } })
16042025-03-25T22:32:03.991ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1037, tv_nsec: 372322942 } })
16052025-03-25T22:32:03.991ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16062025-03-25T22:32:03.991ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
16072025-03-25T22:32:03.991ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
16082025-03-25T22:32:03.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16092025-03-25T22:32:03.993ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16102025-03-25T22:32:03.993ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 11.745254ms iteration = 8 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
16112025-03-25T22:32:03.995ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16122025-03-25T22:32:03.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
16132025-03-25T22:32:03.997ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16142025-03-25T22:32:03.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16152025-03-25T22:32:03.998ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
16162025-03-25T22:32:03.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
16172025-03-25T22:32:03.999ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
16182025-03-25T22:32:03.999ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16192025-03-25T22:32:03.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16202025-03-25T22:32:04.000ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16212025-03-25T22:32:04.002ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16222025-03-25T22:32:04.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
16232025-03-25T22:32:04.002ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16242025-03-25T22:32:04.002ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16252025-03-25T22:32:04.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16262025-03-25T22:32:04.004ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16272025-03-25T22:32:04.006ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
16282025-03-25T22:32:04.006ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16292025-03-25T22:32:04.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16302025-03-25T22:32:04.006ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 23.793184ms iteration = 8 status = {"errors":[],"finish_invoked_ok":[]}
16312025-03-25T22:32:04.007ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
16322025-03-25T22:32:04.007ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
16332025-03-25T22:32:04.007ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16342025-03-25T22:32:04.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
16352025-03-25T22:32:04.007ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
16362025-03-25T22:32:04.007ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16372025-03-25T22:32:04.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16382025-03-25T22:32:04.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16392025-03-25T22:32:04.008ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16402025-03-25T22:32:04.009ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16412025-03-25T22:32:04.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16422025-03-25T22:32:04.011ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
16432025-03-25T22:32:04.011ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
16442025-03-25T22:32:04.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
16452025-03-25T22:32:04.014ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16462025-03-25T22:32:04.014ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16472025-03-25T22:32:04.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16482025-03-25T22:32:04.016ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16492025-03-25T22:32:04.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16502025-03-25T22:32:04.017ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
16512025-03-25T22:32:04.017ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
16522025-03-25T22:32:04.017ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16532025-03-25T22:32:04.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16542025-03-25T22:32:04.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
16552025-03-25T22:32:04.023ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16562025-03-25T22:32:04.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16572025-03-25T22:32:04.024ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1037, tv_nsec: 405148450 } })
16582025-03-25T22:32:04.024ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
16592025-03-25T22:32:04.026ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
16602025-03-25T22:32:04.026ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 43.101746ms iteration = 8 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
16612025-03-25T22:32:04.052ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 69.619959ms iteration = 8 status = {"errors":[],"requests_created_ok":[]}
16622025-03-25T22:32:04.053ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16632025-03-25T22:32:04.053ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
16642025-03-25T22:32:04.053ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16652025-03-25T22:32:04.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
16662025-03-25T22:32:04.056ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16672025-03-25T22:32:04.056ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
16682025-03-25T22:32:04.056ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16692025-03-25T22:32:04.057ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
16702025-03-25T22:32:04.061ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 77.938242ms iteration = 8 status = {"errors":[],"garbage_collect_requested":[]}
16712025-03-25T22:32:04.061ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
16722025-03-25T22:32:04.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
16732025-03-25T22:32:04.064ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16742025-03-25T22:32:04.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16752025-03-25T22:32:04.066ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16762025-03-25T22:32:04.067ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16772025-03-25T22:32:04.067ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16782025-03-25T22:32:04.068ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
16792025-03-25T22:32:04.068ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16802025-03-25T22:32:04.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16812025-03-25T22:32:04.068ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 85.52942ms iteration = 8 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
16822025-03-25T22:32:04.069ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 86.560279ms iteration = 22 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16832025-03-25T22:32:04.069ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16842025-03-25T22:32:04.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16852025-03-25T22:32:04.071ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16862025-03-25T22:32:04.075ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
16872025-03-25T22:32:04.075ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16882025-03-25T22:32:04.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16892025-03-25T22:32:04.077ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 94.200728ms iteration = 8 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
16902025-03-25T22:32:04.078ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 94.997422ms iteration = 8 status = 0
16912025-03-25T22:32:04.081ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
16922025-03-25T22:32:04.081ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16932025-03-25T22:32:04.081ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea1c448e-5a1c-453c-94b9-a4dc1fc0a922", "content-length": "26", "date": "Tue, 25 Mar 2025 22:32:04 GMT"} })
16942025-03-25T22:32:04.081ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16952025-03-25T22:32:04.083ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16962025-03-25T22:32:04.083ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
16972025-03-25T22:32:04.087ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
16982025-03-25T22:32:04.087ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16992025-03-25T22:32:04.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17002025-03-25T22:32:04.092ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
17012025-03-25T22:32:04.092ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
17022025-03-25T22:32:04.093ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2418f573-b039-4390-ac93-8cb0d43b6f3a", "content-length": "26", "date": "Tue, 25 Mar 2025 22:32:04 GMT"} })
17032025-03-25T22:32:04.093ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
17042025-03-25T22:32:04.094ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
17052025-03-25T22:32:04.099ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
17062025-03-25T22:32:04.099ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
17072025-03-25T22:32:04.099ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 116.349032ms iteration = 8 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
17082025-03-25T22:32:04.100ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17092025-03-25T22:32:04.102ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
17102025-03-25T22:32:04.102ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17112025-03-25T22:32:04.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17122025-03-25T22:32:04.104ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
17132025-03-25T22:32:04.104ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 121.106109ms iteration = 22 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
17142025-03-25T22:32:04.105ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 122.066846ms iteration = 8 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
17152025-03-25T22:32:04.120ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
17162025-03-25T22:32:04.120ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 137.849334ms iteration = 8 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
17172025-03-25T22:32:04.149ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
17182025-03-25T22:32:04.149ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
17192025-03-25T22:32:04.149ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb5ebf58-2a4b-47ec-a1b2-5ad3f0e89b01", "content-length": "26", "date": "Tue, 25 Mar 2025 22:32:04 GMT"} })
17202025-03-25T22:32:04.149ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
17212025-03-25T22:32:04.150ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
17222025-03-25T22:32:04.164ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17232025-03-25T22:32:04.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17242025-03-25T22:32:04.167ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17252025-03-25T22:32:04.168ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
17262025-03-25T22:32:04.168ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17272025-03-25T22:32:04.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17282025-03-25T22:32:04.170ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17292025-03-25T22:32:04.173ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
17302025-03-25T22:32:04.173ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "acc5d690-8a51-4525-a3ce-9d8a550c170d", "content-length": "1", "date": "Tue, 25 Mar 2025 22:32:03 GMT"} })
17312025-03-25T22:32:04.174ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
17322025-03-25T22:32:04.174ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17332025-03-25T22:32:04.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17342025-03-25T22:32:04.176ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
17352025-03-25T22:32:04.176ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
17362025-03-25T22:32:04.179ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 197.091194ms iteration = 8 status = null
17372025-03-25T22:32:04.200ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 217.16588ms iteration = 9 status = {"physical_disks_added":0}
17382025-03-25T22:32:04.215ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
17392025-03-25T22:32:04.215ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
17402025-03-25T22:32:04.215ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e10ba39f-bb4d-417a-9eb2-f0a1ea4925cd", "content-length": "372", "date": "Tue, 25 Mar 2025 22:32:04 GMT"} })
17412025-03-25T22:32:04.215ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17422025-03-25T22:32:04.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17432025-03-25T22:32:04.267ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 285.552508ms iteration = 8 status = {}
17442025-03-25T22:32:04.271ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
17452025-03-25T22:32:04.271ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
17462025-03-25T22:32:04.272ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "b163cf72-8d8d-46ae-bb9c-4fda0c1bd73a", "date": "Tue, 25 Mar 2025 22:32:04 GMT"} })
17472025-03-25T22:32:04.272ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17482025-03-25T22:32:04.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17492025-03-25T22:32:04.305ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
17502025-03-25T22:32:04.305ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "740e36b4-8bf1-42a5-b54a-dba6db031193", "content-length": "691", "date": "Tue, 25 Mar 2025 22:32:04 GMT"} })
17512025-03-25T22:32:04.305ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
17522025-03-25T22:32:04.305ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
17532025-03-25T22:32:04.305ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
17542025-03-25T22:32:04.306ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 323.750855ms iteration = 8 status = {}
17552025-03-25T22:32:04.307ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17562025-03-25T22:32:04.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17572025-03-25T22:32:04.310ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17582025-03-25T22:32:04.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17592025-03-25T22:32:04.314ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
17602025-03-25T22:32:04.314ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
17612025-03-25T22:32:04.314ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "82a3d29b-1e58-486c-be2e-aae543b00076", "date": "Tue, 25 Mar 2025 22:32:04 GMT"} })
17622025-03-25T22:32:04.316ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 333.35239ms iteration = 8 status = {}
17632025-03-25T22:32:04.350ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17642025-03-25T22:32:04.351ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17652025-03-25T22:32:04.358ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
17662025-03-25T22:32:04.358ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17672025-03-25T22:32:04.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17682025-03-25T22:32:04.373ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
17692025-03-25T22:32:04.373ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef00ce8f-5268-4fda-91ee-2fcaad7c5909", "content-length": "134", "date": "Tue, 25 Mar 2025 22:32:03 GMT"} })
17702025-03-25T22:32:04.373ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
17712025-03-25T22:32:04.373ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
17722025-03-25T22:32:04.374ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
17732025-03-25T22:32:04.374ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
17742025-03-25T22:32:04.374ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17752025-03-25T22:32:04.374ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17762025-03-25T22:32:04.374ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17772025-03-25T22:32:04.374ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
17782025-03-25T22:32:04.374ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17792025-03-25T22:32:04.374ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
17802025-03-25T22:32:04.374ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
17812025-03-25T22:32:04.374ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17822025-03-25T22:32:04.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17832025-03-25T22:32:04.377ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
17842025-03-25T22:32:04.377ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6926361-cb3e-444d-a33b-ed30d2da42ab", "content-length": "2", "date": "Tue, 25 Mar 2025 22:32:03 GMT"} })
17852025-03-25T22:32:04.378ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
17862025-03-25T22:32:04.378ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bc9dad4-74eb-491f-ae1c-b7fd79566d38", "content-length": "52", "date": "Tue, 25 Mar 2025 22:32:03 GMT"} })
17872025-03-25T22:32:04.378ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
17882025-03-25T22:32:04.378ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
17892025-03-25T22:32:04.409ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
17902025-03-25T22:32:04.409ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
17912025-03-25T22:32:04.452ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
17922025-03-25T22:32:04.452ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:58261
17932025-03-25T22:32:04.452ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:58261 req_id = 6ab6194e-9ff0-4f20-8485-7fca4e3e563e roles = RoleSet { roles: {} } uri = /metrics/producers
17942025-03-25T22:32:04.453ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:58261 req_id = 6ab6194e-9ff0-4f20-8485-7fca4e3e563e resource = Database result = Ok(()) uri = /metrics/producers
17952025-03-25T22:32:04.491ZINFOnexus: assigned collector to new producer collector_id = 26728504-e36d-4baa-856d-cd0ce7a172a2 file = nexus/src/app/oximeter.rs:76 producer_id = 56ee8e9a-ccae-49c2-ac33-94d5159567dd
17962025-03-25T22:32:04.492ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 39200 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:58261 req_id = 6ab6194e-9ff0-4f20-8485-7fca4e3e563e response_code = 201 uri = /metrics/producers
17972025-03-25T22:32:04.492ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "6ab6194e-9ff0-4f20-8485-7fca4e3e563e", "content-length": "41", "date": "Tue, 25 Mar 2025 22:32:04 GMT"} })
17982025-03-25T22:32:04.493ZDEBGnexus (producer-registration-task): registered with nexus successfully
17992025-03-25T22:32:04.493ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
18002025-03-25T22:32:04.577ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "89d31c1f-e42d-46cd-99f5-5690e22a110e", "date": "Tue, 25 Mar 2025 22:32:04 GMT"} })
18012025-03-25T22:32:04.577ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
18022025-03-25T22:32:04.577ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
18032025-03-25T22:32:04.577ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd4551a3-5d74-438f-a42e-a681fd6fa164", "content-length": "682", "date": "Tue, 25 Mar 2025 22:32:04 GMT"} })
18042025-03-25T22:32:04.577ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18052025-03-25T22:32:04.577ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18062025-03-25T22:32:04.581ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18072025-03-25T22:32:04.581ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18082025-03-25T22:32:04.584ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18092025-03-25T22:32:04.585ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18102025-03-25T22:32:04.587ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18112025-03-25T22:32:04.587ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18122025-03-25T22:32:04.593ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 610.536946ms iteration = 8 status = {}
18132025-03-25T22:32:06.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 277 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = cb60c624-c69e-4645-b69e-c63debd528fa response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
18142025-03-25T22:32:13.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 23 reason = Timeout
18152025-03-25T22:32:13.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 23 reason = Timeout
18162025-03-25T22:32:13.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18172025-03-25T22:32:13.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18182025-03-25T22:32:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18192025-03-25T22:32:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18202025-03-25T22:32:13.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18212025-03-25T22:32:13.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18222025-03-25T22:32:13.987ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18232025-03-25T22:32:13.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18242025-03-25T22:32:13.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18252025-03-25T22:32:13.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18262025-03-25T22:32:13.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18272025-03-25T22:32:13.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
18282025-03-25T22:32:13.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18292025-03-25T22:32:13.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18302025-03-25T22:32:14.001ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
18312025-03-25T22:32:14.001ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.192702ms iteration = 23 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
18322025-03-25T22:32:14.020ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18332025-03-25T22:32:14.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18342025-03-25T22:32:14.051ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 67.95205ms iteration = 23 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
18352025-03-25T22:32:16.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 304 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 928f21f2-ec83-401f-abf1-998de760853e response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
18362025-03-25T22:32:23.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 24 reason = Timeout
18372025-03-25T22:32:23.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 24 reason = Timeout
18382025-03-25T22:32:23.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18392025-03-25T22:32:23.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18402025-03-25T22:32:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18412025-03-25T22:32:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18422025-03-25T22:32:23.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18432025-03-25T22:32:23.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18442025-03-25T22:32:23.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18452025-03-25T22:32:23.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.940892ms iteration = 24 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
18462025-03-25T22:32:23.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18472025-03-25T22:32:23.987ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18482025-03-25T22:32:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18492025-03-25T22:32:23.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18502025-03-25T22:32:23.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18512025-03-25T22:32:23.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18522025-03-25T22:32:23.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
18532025-03-25T22:32:23.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18542025-03-25T22:32:23.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18552025-03-25T22:32:24.001ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
18562025-03-25T22:32:24.001ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.11612ms iteration = 24 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
18572025-03-25T22:32:26.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 305 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = c0658984-1572-4f68-b93d-d95964ca90b3 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
18582025-03-25T22:32:31.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 66bc1347-c983-473e-bd3c-4c5145ca865f roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
18592025-03-25T22:32:31.079ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 66bc1347-c983-473e-bd3c-4c5145ca865f resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
18602025-03-25T22:32:31.082ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3595 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 66bc1347-c983-473e-bd3c-4c5145ca865f response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
18612025-03-25T22:32:31.082ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = e85286a4-7db5-4a31-8851-51fafd5ff2c1 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
18622025-03-25T22:32:31.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = e85286a4-7db5-4a31-8851-51fafd5ff2c1 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
18632025-03-25T22:32:31.085ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2983 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = e85286a4-7db5-4a31-8851-51fafd5ff2c1 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
18642025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 5 reason = Timeout
18652025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 9 reason = Timeout
18662025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
18672025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 7 reason = Timeout
18682025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 5 reason = Timeout
18692025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18702025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18712025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 5 reason = Timeout
18722025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 5 reason = Timeout
18732025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18742025-03-25T22:32:33.981ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:22:33.981848230 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
18752025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
18762025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 7 reason = Timeout
18772025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 9 reason = Timeout
18782025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 5 reason = Timeout
18792025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
18802025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18812025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 9 reason = Timeout
18822025-03-25T22:32:33.981ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18832025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 5 reason = Timeout
18842025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18852025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
18862025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18872025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
18882025-03-25T22:32:33.982ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8f1103be-ae21-497e-8fb0-5373f3346d6f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1067, tv_nsec: 363074133 } })
18892025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18902025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
18912025-03-25T22:32:33.982ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("09fed906-08a5-4e51-9e85-fba7211b9594.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4ba21a7b-c053-4a77-807f-34395131fce2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1067, tv_nsec: 363115324 } })
18922025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 9 reason = Timeout
18932025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 9 reason = Timeout
18942025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 9 reason = Timeout
18952025-03-25T22:32:33.982ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
18962025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
18972025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
18982025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18992025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19002025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
19012025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 5 reason = Timeout
19022025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 835.795\u{b5}s iteration = 5 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
19032025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
19042025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 9 reason = Timeout
19052025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 5 reason = Timeout
19062025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
19072025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
19082025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 9 reason = Timeout
19092025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 9 reason = Timeout
19102025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19112025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
19122025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 10 reason = Timeout
19132025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 25 reason = Timeout
19142025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19152025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
19162025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 25 reason = Timeout
19172025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19182025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 9 reason = Timeout
19192025-03-25T22:32:33.982ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19202025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 9 reason = Timeout
19212025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
19222025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 9 reason = Timeout
19232025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
19242025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 9 reason = Timeout
19252025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19262025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 9 reason = Timeout
19272025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
19282025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 9 reason = Timeout
19292025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
19302025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 5 reason = Timeout
19312025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
19322025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 9 reason = Timeout
19332025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
19342025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 5 reason = Timeout
19352025-03-25T22:32:33.983ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
19362025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
19372025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 6 reason = Timeout
19382025-03-25T22:32:33.983ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b
19392025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
19402025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 12.93\u{b5}s iteration = 6 status = {"enabled":false,"target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b"}
19412025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19422025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19432025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
19442025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
19452025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19462025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
19472025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19482025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19492025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
19502025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.586299ms iteration = 5 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
19512025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19522025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
19532025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19542025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19552025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
19562025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
19572025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
19582025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
19592025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
19602025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
19612025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
19622025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19632025-03-25T22:32:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
19642025-03-25T22:32:33.985ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.270949ms iteration = 9 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
19652025-03-25T22:32:33.985ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19662025-03-25T22:32:33.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19672025-03-25T22:32:33.985ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19682025-03-25T22:32:33.985ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:22:33.981848230 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
19692025-03-25T22:32:33.985ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.064705ms iteration = 5 status = {"expiration":"2025-03-25T22:22:33.981848230Z","pruned":[]}
19702025-03-25T22:32:33.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1067, tv_nsec: 367022605 } })
19712025-03-25T22:32:33.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
19722025-03-25T22:32:33.987ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19732025-03-25T22:32:33.988ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1067, tv_nsec: 368775547 } })
19742025-03-25T22:32:33.988ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
19752025-03-25T22:32:33.989ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
19762025-03-25T22:32:33.989ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19772025-03-25T22:32:33.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19782025-03-25T22:32:33.990ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
19792025-03-25T22:32:33.990ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19802025-03-25T22:32:33.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19812025-03-25T22:32:33.993ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19822025-03-25T22:32:33.994ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19832025-03-25T22:32:33.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19842025-03-25T22:32:33.997ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 14.441104ms iteration = 5 status = {"errors":[],"found_port_ok":[]}
19852025-03-25T22:32:33.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
19862025-03-25T22:32:33.997ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19872025-03-25T22:32:33.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19882025-03-25T22:32:33.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
19892025-03-25T22:32:33.998ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19902025-03-25T22:32:33.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19912025-03-25T22:32:33.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
19922025-03-25T22:32:33.999ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19932025-03-25T22:32:33.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19942025-03-25T22:32:34.007ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19952025-03-25T22:32:34.010ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
19962025-03-25T22:32:34.010ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19972025-03-25T22:32:34.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19982025-03-25T22:32:34.016ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19992025-03-25T22:32:34.018ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20002025-03-25T22:32:34.020ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
20012025-03-25T22:32:34.020ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20022025-03-25T22:32:34.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20032025-03-25T22:32:34.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
20042025-03-25T22:32:34.021ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
20052025-03-25T22:32:34.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
20062025-03-25T22:32:34.022ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20072025-03-25T22:32:34.023ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20082025-03-25T22:32:34.027ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
20092025-03-25T22:32:34.027ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
20102025-03-25T22:32:34.027ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20112025-03-25T22:32:34.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
20122025-03-25T22:32:34.032ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20132025-03-25T22:32:34.032ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20142025-03-25T22:32:34.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20152025-03-25T22:32:34.036ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20162025-03-25T22:32:34.040ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20172025-03-25T22:32:34.040ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20182025-03-25T22:32:34.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20192025-03-25T22:32:34.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20202025-03-25T22:32:34.041ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20212025-03-25T22:32:34.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20222025-03-25T22:32:34.042ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1067, tv_nsec: 422985566 } })
20232025-03-25T22:32:34.042ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
20242025-03-25T22:32:34.045ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20252025-03-25T22:32:34.050ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
20262025-03-25T22:32:34.050ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
20272025-03-25T22:32:34.050ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20282025-03-25T22:32:34.050ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
20292025-03-25T22:32:34.050ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
20302025-03-25T22:32:34.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20312025-03-25T22:32:34.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
20322025-03-25T22:32:34.060ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20332025-03-25T22:32:34.064ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
20342025-03-25T22:32:34.064ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20352025-03-25T22:32:34.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20362025-03-25T22:32:34.066ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 83.109605ms iteration = 9 status = {"errors":[],"requests_created_ok":[]}
20372025-03-25T22:32:34.072ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20382025-03-25T22:32:34.076ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
20392025-03-25T22:32:34.076ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20402025-03-25T22:32:34.076ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 93.382352ms iteration = 9 status = {"errors":[],"finish_invoked_ok":[]}
20412025-03-25T22:32:34.076ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
20422025-03-25T22:32:34.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20432025-03-25T22:32:34.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
20442025-03-25T22:32:34.079ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20452025-03-25T22:32:34.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20462025-03-25T22:32:34.085ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
20472025-03-25T22:32:34.085ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
20482025-03-25T22:32:34.085ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 102.49436ms iteration = 5 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
20492025-03-25T22:32:34.087ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 104.14346ms iteration = 9 status = {"errors":[],"garbage_collect_requested":[]}
20502025-03-25T22:32:34.087ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
20512025-03-25T22:32:34.087ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
20522025-03-25T22:32:34.087ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
20532025-03-25T22:32:34.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
20542025-03-25T22:32:34.088ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
20552025-03-25T22:32:34.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
20562025-03-25T22:32:34.092ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
20572025-03-25T22:32:34.092ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
20582025-03-25T22:32:34.092ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
20592025-03-25T22:32:34.092ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20602025-03-25T22:32:34.092ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20612025-03-25T22:32:34.093ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20622025-03-25T22:32:34.093ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 110.402523ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
20632025-03-25T22:32:34.095ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
20642025-03-25T22:32:34.095ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20652025-03-25T22:32:34.096ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20662025-03-25T22:32:34.098ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20672025-03-25T22:32:34.099ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
20682025-03-25T22:32:34.100ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
20692025-03-25T22:32:34.100ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
20702025-03-25T22:32:34.100ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 117.30643ms iteration = 25 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
20712025-03-25T22:32:34.103ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20722025-03-25T22:32:34.103ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20732025-03-25T22:32:34.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20742025-03-25T22:32:34.106ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
20752025-03-25T22:32:34.106ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 0 version = 2
20762025-03-25T22:32:34.106ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20772025-03-25T22:32:34.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20782025-03-25T22:32:34.112ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20792025-03-25T22:32:34.113ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 129.897589ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
20802025-03-25T22:32:34.115ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
20812025-03-25T22:32:34.115ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
20822025-03-25T22:32:34.117ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20832025-03-25T22:32:34.117ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20842025-03-25T22:32:34.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20852025-03-25T22:32:34.123ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
20862025-03-25T22:32:34.123ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
20872025-03-25T22:32:34.123ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
20882025-03-25T22:32:34.123ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20892025-03-25T22:32:34.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20902025-03-25T22:32:34.124ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20912025-03-25T22:32:34.124ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20922025-03-25T22:32:34.125ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
20932025-03-25T22:32:34.125ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
20942025-03-25T22:32:34.125ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 141.931878ms iteration = 5 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
20952025-03-25T22:32:34.130ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
20962025-03-25T22:32:34.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
20972025-03-25T22:32:34.134ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
20982025-03-25T22:32:34.134ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 151.644485ms iteration = 9 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
20992025-03-25T22:32:34.138ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21002025-03-25T22:32:34.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21012025-03-25T22:32:34.139ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
21022025-03-25T22:32:34.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
21032025-03-25T22:32:34.141ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 158.850327ms iteration = 25 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
21042025-03-25T22:32:34.147ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
21052025-03-25T22:32:34.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
21062025-03-25T22:32:34.150ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
21072025-03-25T22:32:34.150ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21082025-03-25T22:32:34.150ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
21092025-03-25T22:32:34.150ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
21102025-03-25T22:32:34.150ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 167.828291ms iteration = 9 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
21112025-03-25T22:32:34.150ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e517f004-6f0b-49d1-abac-4b118058efc4", "content-length": "26", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} })
21122025-03-25T22:32:34.150ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21132025-03-25T22:32:34.152ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
21142025-03-25T22:32:34.207ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 224.25642ms iteration = 10 status = {"physical_disks_added":0}
21152025-03-25T22:32:34.219ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
21162025-03-25T22:32:34.250ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
21172025-03-25T22:32:34.251ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21182025-03-25T22:32:34.251ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ea5ef9a-e552-45e2-b3e3-2b8b7ac36b6e", "content-length": "26", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} })
21192025-03-25T22:32:34.251ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21202025-03-25T22:32:34.253ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
21212025-03-25T22:32:34.265ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
21222025-03-25T22:32:34.279ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
21232025-03-25T22:32:34.279ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21242025-03-25T22:32:34.280ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "40dc5544-a78a-428d-852f-043daea02c9a", "content-length": "26", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} })
21252025-03-25T22:32:34.280ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21262025-03-25T22:32:34.281ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
21272025-03-25T22:32:34.330ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
21282025-03-25T22:32:34.330ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
21292025-03-25T22:32:34.330ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "87515456-5dc2-48e4-9c6d-6bdfc58c713a", "content-length": "372", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} })
21302025-03-25T22:32:34.330ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21312025-03-25T22:32:34.331ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21322025-03-25T22:32:34.332ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21332025-03-25T22:32:34.333ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21342025-03-25T22:32:34.334ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 1 version = 2
21352025-03-25T22:32:34.334ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
21362025-03-25T22:32:34.334ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:29:07.710706 UTC dns_group = external generation = 2
21372025-03-25T22:32:34.334ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 352.301085ms iteration = 5 status = {"generation":2}
21382025-03-25T22:32:34.335ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21392025-03-25T22:32:34.335ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 353.422056ms iteration = 9 status = {}
21402025-03-25T22:32:34.335ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21412025-03-25T22:32:34.337ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
21422025-03-25T22:32:34.337ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
21432025-03-25T22:32:34.337ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
21442025-03-25T22:32:34.338ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
21452025-03-25T22:32:34.338ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21462025-03-25T22:32:34.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21472025-03-25T22:32:34.341ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21482025-03-25T22:32:34.346ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
21492025-03-25T22:32:34.346ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21502025-03-25T22:32:34.346ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21512025-03-25T22:32:34.350ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 367.551343ms iteration = 9 status = 0
21522025-03-25T22:32:34.397ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21532025-03-25T22:32:34.403ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
21542025-03-25T22:32:34.403ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21552025-03-25T22:32:34.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21562025-03-25T22:32:34.403ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 420.592501ms iteration = 9 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
21572025-03-25T22:32:34.406ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
21582025-03-25T22:32:34.406ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 423.731268ms iteration = 9 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
21592025-03-25T22:32:34.412ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
21602025-03-25T22:32:34.415ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
21612025-03-25T22:32:34.415ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
21622025-03-25T22:32:34.415ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
21632025-03-25T22:32:34.415ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21642025-03-25T22:32:34.415ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "529c4605-a54f-4ab0-abb1-eb64e5662695", "content-length": "1", "date": "Tue, 25 Mar 2025 22:32:33 GMT"} })
21652025-03-25T22:32:34.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21662025-03-25T22:32:34.417ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
21672025-03-25T22:32:34.417ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "9732ef18-15e0-4ca6-b4cd-5f431aca7761", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
21682025-03-25T22:32:34.418ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
21692025-03-25T22:32:34.418ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
21702025-03-25T22:32:34.418ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
21712025-03-25T22:32:34.418ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21722025-03-25T22:32:34.420ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "4bf68a7d-a9b7-45cb-987f-28207c8cecd0", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
21732025-03-25T22:32:34.420ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
21742025-03-25T22:32:34.421ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
21752025-03-25T22:32:34.421ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
21762025-03-25T22:32:34.421ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 439.580587ms iteration = 9 status = null
21772025-03-25T22:32:34.421ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "f2134601-48bd-4317-9344-23ac3e6278fb", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} })
21782025-03-25T22:32:34.421ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21792025-03-25T22:32:34.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21802025-03-25T22:32:34.422ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
21812025-03-25T22:32:34.422ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 440.772268ms iteration = 7 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
21822025-03-25T22:32:34.423ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
21832025-03-25T22:32:34.423ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21842025-03-25T22:32:34.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21852025-03-25T22:32:34.425ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
21862025-03-25T22:32:34.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
21872025-03-25T22:32:34.426ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21882025-03-25T22:32:34.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21892025-03-25T22:32:34.427ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21902025-03-25T22:32:34.429ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
21912025-03-25T22:32:34.429ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
21922025-03-25T22:32:34.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
21932025-03-25T22:32:34.430ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21942025-03-25T22:32:34.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21952025-03-25T22:32:34.432ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
21962025-03-25T22:32:34.432ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
21972025-03-25T22:32:34.432ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:29:06.674111 UTC dns_group = internal generation = 1
21982025-03-25T22:32:34.432ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 450.830703ms iteration = 5 status = {"generation":1}
21992025-03-25T22:32:34.434ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
22002025-03-25T22:32:34.434ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
22012025-03-25T22:32:34.434ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "14a00929-41d6-4ca7-919a-926e7856aca6", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} })
22022025-03-25T22:32:34.437ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 454.427268ms iteration = 9 status = {}
22032025-03-25T22:32:34.455ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:28:31.652727Z, time_modified: 2025-03-25T22:28:31.652727Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }) }
22042025-03-25T22:32:34.455ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 473.215071ms iteration = 5 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]}},"default_endpoint":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
22052025-03-25T22:32:34.488ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22062025-03-25T22:32:34.489ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
22072025-03-25T22:32:34.490ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "d9105759-63e6-496c-9ef7-b157503d1b08", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22082025-03-25T22:32:34.490ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22092025-03-25T22:32:34.491ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "48a3763f-b560-4969-8738-fe19dc916ef9", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22102025-03-25T22:32:34.491ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22112025-03-25T22:32:34.492ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
22122025-03-25T22:32:34.492ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22132025-03-25T22:32:34.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22142025-03-25T22:32:34.493ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "1c3fad3a-a3b7-4a50-b889-700dcf99edfd", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22152025-03-25T22:32:34.493ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22162025-03-25T22:32:34.496ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22172025-03-25T22:32:34.496ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 514.588455ms iteration = 7 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
22182025-03-25T22:32:34.517ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
22192025-03-25T22:32:34.517ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 534.55969ms iteration = 5 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
22202025-03-25T22:32:34.546ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22212025-03-25T22:32:34.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22222025-03-25T22:32:34.549ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
22232025-03-25T22:32:34.549ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22242025-03-25T22:32:34.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22252025-03-25T22:32:34.552ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
22262025-03-25T22:32:34.552ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "5feaf027-b2dd-4a66-b233-dbd7ae210d9a", "content-length": "691", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} })
22272025-03-25T22:32:34.552ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
22282025-03-25T22:32:34.552ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
22292025-03-25T22:32:34.552ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
22302025-03-25T22:32:34.554ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 571.380411ms iteration = 9 status = {}
22312025-03-25T22:32:34.559ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
22322025-03-25T22:32:34.559ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "0edddf98-7e83-46e0-9a97-1047599c34a7", "content-length": "134", "date": "Tue, 25 Mar 2025 22:32:33 GMT"} })
22332025-03-25T22:32:34.559ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
22342025-03-25T22:32:34.559ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
22352025-03-25T22:32:34.560ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
22362025-03-25T22:32:34.560ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
22372025-03-25T22:32:34.560ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22382025-03-25T22:32:34.560ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22392025-03-25T22:32:34.560ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22402025-03-25T22:32:34.560ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
22412025-03-25T22:32:34.560ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22422025-03-25T22:32:34.560ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
22432025-03-25T22:32:34.560ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
22442025-03-25T22:32:34.560ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22452025-03-25T22:32:34.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22462025-03-25T22:32:34.562ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
22472025-03-25T22:32:34.563ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "caee4501-c6da-454f-8473-d6ad923fc2f2", "content-length": "2", "date": "Tue, 25 Mar 2025 22:32:33 GMT"} })
22482025-03-25T22:32:34.563ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
22492025-03-25T22:32:34.563ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fe8d20d-4881-4579-a501-57179df11af6", "content-length": "52", "date": "Tue, 25 Mar 2025 22:32:33 GMT"} })
22502025-03-25T22:32:34.563ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
22512025-03-25T22:32:34.563ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
22522025-03-25T22:32:34.661ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "fd209484-6cd1-47d7-874b-9728019699e8", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} })
22532025-03-25T22:32:34.661ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
22542025-03-25T22:32:34.661ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
22552025-03-25T22:32:34.661ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "b608f5f6-d694-4e23-9c27-cc07cea930b8", "content-length": "682", "date": "Tue, 25 Mar 2025 22:32:34 GMT"} })
22562025-03-25T22:32:34.661ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22572025-03-25T22:32:34.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22582025-03-25T22:32:34.665ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22592025-03-25T22:32:34.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22602025-03-25T22:32:34.669ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22612025-03-25T22:32:34.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22622025-03-25T22:32:34.672ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22632025-03-25T22:32:34.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22642025-03-25T22:32:34.678ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 695.879802ms iteration = 9 status = {}
22652025-03-25T22:32:36.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 297 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 82bf54c1-fe0c-4332-a3a3-3086f1cb9f2f response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
22662025-03-25T22:32:43.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 26 reason = Timeout
22672025-03-25T22:32:43.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 26 reason = Timeout
22682025-03-25T22:32:43.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22692025-03-25T22:32:43.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22702025-03-25T22:32:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22712025-03-25T22:32:43.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22722025-03-25T22:32:43.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22732025-03-25T22:32:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
22742025-03-25T22:32:43.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22752025-03-25T22:32:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22762025-03-25T22:32:43.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22772025-03-25T22:32:43.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22782025-03-25T22:32:43.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22792025-03-25T22:32:43.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
22802025-03-25T22:32:43.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22812025-03-25T22:32:43.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22822025-03-25T22:32:44.006ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
22832025-03-25T22:32:44.006ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.843826ms iteration = 26 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
22842025-03-25T22:32:44.015ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22852025-03-25T22:32:44.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22862025-03-25T22:32:44.017ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.673034ms iteration = 26 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
22872025-03-25T22:32:46.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 273 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 9cf0af4e-7416-4bef-a424-e4e674a50acc response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
22882025-03-25T22:32:53.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 27 reason = Timeout
22892025-03-25T22:32:53.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 27 reason = Timeout
22902025-03-25T22:32:53.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22912025-03-25T22:32:53.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22922025-03-25T22:32:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22932025-03-25T22:32:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22942025-03-25T22:32:53.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22952025-03-25T22:32:53.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22962025-03-25T22:32:53.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22972025-03-25T22:32:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
22982025-03-25T22:32:53.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22992025-03-25T22:32:53.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23002025-03-25T22:32:53.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23012025-03-25T22:32:53.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23022025-03-25T22:32:53.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23032025-03-25T22:32:53.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
23042025-03-25T22:32:53.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23052025-03-25T22:32:53.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23062025-03-25T22:32:54.004ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
23072025-03-25T22:32:54.004ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.36039ms iteration = 27 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
23082025-03-25T22:32:54.015ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.669277ms iteration = 27 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
23092025-03-25T22:32:56.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 250 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 273436a9-320d-4529-bea6-8573f406170e response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
23102025-03-25T22:33:01.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 70c6d3ce-c004-43c2-9b36-94dda2bc0df1 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
23112025-03-25T22:33:01.078ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 70c6d3ce-c004-43c2-9b36-94dda2bc0df1 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
23122025-03-25T22:33:01.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3755 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 70c6d3ce-c004-43c2-9b36-94dda2bc0df1 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
23132025-03-25T22:33:01.082ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = cdbc9153-2761-4507-a41b-dc45d66d06bb roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
23142025-03-25T22:33:01.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = cdbc9153-2761-4507-a41b-dc45d66d06bb resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
23152025-03-25T22:33:01.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2728 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = cdbc9153-2761-4507-a41b-dc45d66d06bb response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
23162025-03-25T22:33:03.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 10 reason = Timeout
23172025-03-25T22:33:03.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 10 reason = Timeout
23182025-03-25T22:33:03.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 10 reason = Timeout
23192025-03-25T22:33:03.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
23202025-03-25T22:33:03.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
23212025-03-25T22:33:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
23222025-03-25T22:33:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
23232025-03-25T22:33:03.982ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 10 reason = Timeout
23242025-03-25T22:33:03.982ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23252025-03-25T22:33:03.982ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 28 reason = Timeout
23262025-03-25T22:33:03.982ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 10 reason = Timeout
23272025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23282025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
23292025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 10 reason = Timeout
23302025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23312025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 11 reason = Timeout
23322025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23332025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 10 reason = Timeout
23342025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 10 reason = Timeout
23352025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
23362025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23372025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 10 reason = Timeout
23382025-03-25T22:33:03.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
23392025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23402025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 10 reason = Timeout
23412025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 10 reason = Timeout
23422025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
23432025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
23442025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 10 reason = Timeout
23452025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
23462025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 10 reason = Timeout
23472025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23482025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 10 reason = Timeout
23492025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
23502025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
23512025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 28 reason = Timeout
23522025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23532025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 10 reason = Timeout
23542025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
23552025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 10 reason = Timeout
23562025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23572025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
23582025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23592025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23602025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23612025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23622025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
23632025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
23642025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
23652025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23662025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
23672025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
23682025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23692025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
23702025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
23712025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23722025-03-25T22:33:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
23732025-03-25T22:33:03.985ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23742025-03-25T22:33:03.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1097, tv_nsec: 366500536 } })
23752025-03-25T22:33:03.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1097, tv_nsec: 366765040 } })
23762025-03-25T22:33:03.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
23772025-03-25T22:33:03.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23782025-03-25T22:33:03.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
23792025-03-25T22:33:03.986ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.510122ms iteration = 10 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
23802025-03-25T22:33:03.986ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23812025-03-25T22:33:03.987ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23822025-03-25T22:33:03.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
23832025-03-25T22:33:03.988ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23842025-03-25T22:33:03.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23852025-03-25T22:33:03.990ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
23862025-03-25T22:33:03.990ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23872025-03-25T22:33:03.990ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
23882025-03-25T22:33:03.990ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23892025-03-25T22:33:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23902025-03-25T22:33:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23912025-03-25T22:33:03.991ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23922025-03-25T22:33:03.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
23932025-03-25T22:33:03.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23942025-03-25T22:33:03.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23952025-03-25T22:33:03.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
23962025-03-25T22:33:03.994ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
23972025-03-25T22:33:03.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
23982025-03-25T22:33:03.995ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23992025-03-25T22:33:03.996ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24002025-03-25T22:33:03.997ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 14.507294ms iteration = 10 status = {"errors":[],"garbage_collect_requested":[]}
24012025-03-25T22:33:03.997ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
24022025-03-25T22:33:03.997ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
24032025-03-25T22:33:03.997ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
24042025-03-25T22:33:03.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
24052025-03-25T22:33:03.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
24062025-03-25T22:33:03.998ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24072025-03-25T22:33:03.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24082025-03-25T22:33:04.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
24092025-03-25T22:33:04.001ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24102025-03-25T22:33:04.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24112025-03-25T22:33:04.005ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24122025-03-25T22:33:04.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24132025-03-25T22:33:04.012ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24142025-03-25T22:33:04.012ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
24152025-03-25T22:33:04.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24162025-03-25T22:33:04.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
24172025-03-25T22:33:04.018ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24182025-03-25T22:33:04.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24192025-03-25T22:33:04.019ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 36.757821ms iteration = 28 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
24202025-03-25T22:33:04.021ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24212025-03-25T22:33:04.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24222025-03-25T22:33:04.022ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1097, tv_nsec: 403169214 } })
24232025-03-25T22:33:04.022ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
24242025-03-25T22:33:04.041ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24252025-03-25T22:33:04.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24262025-03-25T22:33:04.044ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24272025-03-25T22:33:04.044ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 61.090874ms iteration = 10 status = {"errors":[],"finish_invoked_ok":[]}
24282025-03-25T22:33:04.047ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
24292025-03-25T22:33:04.047ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24302025-03-25T22:33:04.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24312025-03-25T22:33:04.048ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
24322025-03-25T22:33:04.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
24332025-03-25T22:33:04.049ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24342025-03-25T22:33:04.051ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24352025-03-25T22:33:04.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24362025-03-25T22:33:04.052ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
24372025-03-25T22:33:04.052ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24382025-03-25T22:33:04.052ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24392025-03-25T22:33:04.053ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
24402025-03-25T22:33:04.053ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24412025-03-25T22:33:04.053ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "14530379-dfef-47c3-95fa-5d327c514334", "content-length": "26", "date": "Tue, 25 Mar 2025 22:33:04 GMT"} })
24422025-03-25T22:33:04.053ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24432025-03-25T22:33:04.053ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24442025-03-25T22:33:04.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24452025-03-25T22:33:04.054ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
24462025-03-25T22:33:04.062ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24472025-03-25T22:33:04.064ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
24482025-03-25T22:33:04.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
24492025-03-25T22:33:04.065ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
24502025-03-25T22:33:04.065ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24512025-03-25T22:33:04.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24522025-03-25T22:33:04.068ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
24532025-03-25T22:33:04.068ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24542025-03-25T22:33:04.068ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "55659142-b1ab-44f9-b2a8-2a66b4e357de", "content-length": "26", "date": "Tue, 25 Mar 2025 22:33:04 GMT"} })
24552025-03-25T22:33:04.068ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24562025-03-25T22:33:04.070ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
24572025-03-25T22:33:04.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
24582025-03-25T22:33:04.070ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
24592025-03-25T22:33:04.070ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 87.410695ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
24602025-03-25T22:33:04.078ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
24612025-03-25T22:33:04.078ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
24622025-03-25T22:33:04.078ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
24632025-03-25T22:33:04.078ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24642025-03-25T22:33:04.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
24652025-03-25T22:33:04.080ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 97.278304ms iteration = 10 status = {"errors":[],"requests_created_ok":[]}
24662025-03-25T22:33:04.080ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24672025-03-25T22:33:04.081ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
24682025-03-25T22:33:04.081ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
24692025-03-25T22:33:04.081ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
24702025-03-25T22:33:04.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
24712025-03-25T22:33:04.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
24722025-03-25T22:33:04.083ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
24732025-03-25T22:33:04.083ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 100.039275ms iteration = 10 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
24742025-03-25T22:33:04.084ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
24752025-03-25T22:33:04.084ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24762025-03-25T22:33:04.084ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 101.470291ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
24772025-03-25T22:33:04.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24782025-03-25T22:33:04.099ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
24792025-03-25T22:33:04.099ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 116.817291ms iteration = 28 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
24802025-03-25T22:33:04.102ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
24812025-03-25T22:33:04.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
24822025-03-25T22:33:04.109ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24832025-03-25T22:33:04.112ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
24842025-03-25T22:33:04.112ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24852025-03-25T22:33:04.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24862025-03-25T22:33:04.115ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 132.237252ms iteration = 10 status = 0
24872025-03-25T22:33:04.120ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
24882025-03-25T22:33:04.120ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
24892025-03-25T22:33:04.120ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 137.398596ms iteration = 10 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
24902025-03-25T22:33:04.121ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24912025-03-25T22:33:04.124ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
24922025-03-25T22:33:04.124ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
24932025-03-25T22:33:04.125ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
24942025-03-25T22:33:04.155ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 172.046338ms iteration = 10 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
24952025-03-25T22:33:04.175ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
24962025-03-25T22:33:04.175ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 192.46354ms iteration = 10 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
24972025-03-25T22:33:04.206ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 223.788882ms iteration = 11 status = {"physical_disks_added":0}
24982025-03-25T22:33:04.252ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
24992025-03-25T22:33:04.252ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
25002025-03-25T22:33:04.253ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "16cb18a2-ef3c-40ec-8da3-e5399ec878e7", "content-length": "26", "date": "Tue, 25 Mar 2025 22:33:04 GMT"} })
25012025-03-25T22:33:04.253ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
25022025-03-25T22:33:04.254ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
25032025-03-25T22:33:04.314ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
25042025-03-25T22:33:04.315ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "320091be-6635-48f6-bb04-85601ba3793c", "content-length": "691", "date": "Tue, 25 Mar 2025 22:33:04 GMT"} })
25052025-03-25T22:33:04.315ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
25062025-03-25T22:33:04.315ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
25072025-03-25T22:33:04.315ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
25082025-03-25T22:33:04.316ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 333.982111ms iteration = 10 status = {}
25092025-03-25T22:33:04.318ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
25102025-03-25T22:33:04.318ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
25112025-03-25T22:33:04.318ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "934a6c3d-77a7-438b-8909-878a4cae133e", "content-length": "372", "date": "Tue, 25 Mar 2025 22:33:04 GMT"} })
25122025-03-25T22:33:04.318ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25132025-03-25T22:33:04.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25142025-03-25T22:33:04.323ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 341.605481ms iteration = 10 status = {}
25152025-03-25T22:33:04.336ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
25162025-03-25T22:33:04.340ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "f090f309-3bdb-4809-8ed9-8b0f4c983011", "content-length": "1", "date": "Tue, 25 Mar 2025 22:33:03 GMT"} })
25172025-03-25T22:33:04.343ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
25182025-03-25T22:33:04.344ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
25192025-03-25T22:33:04.346ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 364.327845ms iteration = 10 status = null
25202025-03-25T22:33:04.407ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
25212025-03-25T22:33:04.407ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
25222025-03-25T22:33:04.465ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25232025-03-25T22:33:04.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25242025-03-25T22:33:04.468ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
25252025-03-25T22:33:04.468ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25262025-03-25T22:33:04.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25272025-03-25T22:33:04.478ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "2d73e2ad-2593-405a-b8eb-02672f167830", "date": "Tue, 25 Mar 2025 22:33:04 GMT"} })
25282025-03-25T22:33:04.478ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25292025-03-25T22:33:04.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25302025-03-25T22:33:04.482ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25312025-03-25T22:33:04.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25322025-03-25T22:33:04.483ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
25332025-03-25T22:33:04.484ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0de11c9-888b-45b1-bfa5-5867d948ccf2", "content-length": "134", "date": "Tue, 25 Mar 2025 22:33:03 GMT"} })
25342025-03-25T22:33:04.484ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
25352025-03-25T22:33:04.484ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
25362025-03-25T22:33:04.484ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
25372025-03-25T22:33:04.484ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
25382025-03-25T22:33:04.484ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25392025-03-25T22:33:04.484ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25402025-03-25T22:33:04.484ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25412025-03-25T22:33:04.484ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
25422025-03-25T22:33:04.484ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25432025-03-25T22:33:04.484ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
25442025-03-25T22:33:04.485ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
25452025-03-25T22:33:04.485ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25462025-03-25T22:33:04.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25472025-03-25T22:33:04.486ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25482025-03-25T22:33:04.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25492025-03-25T22:33:04.487ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
25502025-03-25T22:33:04.488ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "1de29cb1-0041-4afe-bb4e-67c134a85fa6", "content-length": "2", "date": "Tue, 25 Mar 2025 22:33:03 GMT"} })
25512025-03-25T22:33:04.488ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
25522025-03-25T22:33:04.488ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "26e7e183-77c4-4831-a7fc-c411432a6d9b", "content-length": "52", "date": "Tue, 25 Mar 2025 22:33:03 GMT"} })
25532025-03-25T22:33:04.488ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
25542025-03-25T22:33:04.488ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
25552025-03-25T22:33:04.490ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
25562025-03-25T22:33:04.490ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
25572025-03-25T22:33:04.623ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "a41d1bd5-b10b-4809-bc03-2eaf0c53c3ef", "date": "Tue, 25 Mar 2025 22:33:04 GMT"} })
25582025-03-25T22:33:04.623ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
25592025-03-25T22:33:04.623ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
25602025-03-25T22:33:04.626ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3821542-7f82-4d62-9cf5-d936b004f710", "content-length": "682", "date": "Tue, 25 Mar 2025 22:33:04 GMT"} })
25612025-03-25T22:33:04.626ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25622025-03-25T22:33:04.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25632025-03-25T22:33:04.626ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "fa0fa647-384f-4d3c-aa3c-8a7bd53b12ea", "date": "Tue, 25 Mar 2025 22:33:04 GMT"} })
25642025-03-25T22:33:04.629ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 645.973082ms iteration = 10 status = {}
25652025-03-25T22:33:04.629ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25662025-03-25T22:33:04.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25672025-03-25T22:33:04.633ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25682025-03-25T22:33:04.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25692025-03-25T22:33:04.635ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25702025-03-25T22:33:04.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25712025-03-25T22:33:04.643ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 660.789722ms iteration = 10 status = {}
25722025-03-25T22:33:06.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 292 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 646a02b5-3e63-4e61-b683-8d6255496acf response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
25732025-03-25T22:33:13.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 29 reason = Timeout
25742025-03-25T22:33:13.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
25752025-03-25T22:33:13.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25762025-03-25T22:33:13.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25772025-03-25T22:33:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25782025-03-25T22:33:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25792025-03-25T22:33:13.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25802025-03-25T22:33:13.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25812025-03-25T22:33:13.987ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25822025-03-25T22:33:13.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25832025-03-25T22:33:13.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25842025-03-25T22:33:13.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25852025-03-25T22:33:13.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25862025-03-25T22:33:13.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
25872025-03-25T22:33:13.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25882025-03-25T22:33:13.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25892025-03-25T22:33:14.003ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
25902025-03-25T22:33:14.003ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.330441ms iteration = 29 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
25912025-03-25T22:33:14.013ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25922025-03-25T22:33:14.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25932025-03-25T22:33:14.043ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 60.420551ms iteration = 29 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
25942025-03-25T22:33:16.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 286 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 164ce321-c8cc-4301-a87a-72d899fa1436 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
25952025-03-25T22:33:23.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
25962025-03-25T22:33:23.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 30 reason = Timeout
25972025-03-25T22:33:23.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25982025-03-25T22:33:23.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25992025-03-25T22:33:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26002025-03-25T22:33:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26012025-03-25T22:33:23.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26022025-03-25T22:33:23.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26032025-03-25T22:33:23.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26042025-03-25T22:33:23.986ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.428412ms iteration = 30 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
26052025-03-25T22:33:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
26062025-03-25T22:33:23.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26072025-03-25T22:33:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26082025-03-25T22:33:23.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26092025-03-25T22:33:23.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26102025-03-25T22:33:23.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26112025-03-25T22:33:23.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
26122025-03-25T22:33:23.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26132025-03-25T22:33:23.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26142025-03-25T22:33:24.003ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
26152025-03-25T22:33:24.003ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.131767ms iteration = 30 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
26162025-03-25T22:33:26.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 432 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 7066e5d8-98c3-4b0e-ab50-dc9500259e01 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
26172025-03-25T22:33:31.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 81ff6d2d-6e94-4131-94ab-d14e13966482 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
26182025-03-25T22:33:31.078ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 81ff6d2d-6e94-4131-94ab-d14e13966482 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
26192025-03-25T22:33:31.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3266 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 81ff6d2d-6e94-4131-94ab-d14e13966482 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
26202025-03-25T22:33:31.082ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 8bab6023-b9cf-48e1-ac84-469756066488 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
26212025-03-25T22:33:31.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 8bab6023-b9cf-48e1-ac84-469756066488 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
26222025-03-25T22:33:31.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2510 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 8bab6023-b9cf-48e1-ac84-469756066488 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
26232025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
26242025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
26252025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 8 reason = Timeout
26262025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
26272025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
26282025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 6 reason = Timeout
26292025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
26302025-03-25T22:33:33.982ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:23:33.982164505 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
26312025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
26322025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
26332025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
26342025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
26352025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
26362025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
26372025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
26382025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
26392025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 8 reason = Timeout
26402025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
26412025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
26422025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
26432025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
26442025-03-25T22:33:33.982ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("09fed906-08a5-4e51-9e85-fba7211b9594.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4ba21a7b-c053-4a77-807f-34395131fce2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1127, tv_nsec: 363235576 } })
26452025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
26462025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
26472025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
26482025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
26492025-03-25T22:33:33.982ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8f1103be-ae21-497e-8fb0-5373f3346d6f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1127, tv_nsec: 363418959 } })
26502025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 11 reason = Timeout
26512025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
26522025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
26532025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
26542025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 6 reason = Timeout
26552025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 2 reason = Timeout
26562025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 6 reason = Timeout
26572025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
26582025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 3 reason = Timeout
26592025-03-25T22:33:33.982ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
26602025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 11 reason = Timeout
26612025-03-25T22:33:33.982ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
26622025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
26632025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 6 reason = Timeout
26642025-03-25T22:33:33.982ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
26652025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 11 reason = Timeout
26662025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
26672025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
26682025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26692025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
26702025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
26712025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
26722025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26732025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 12 reason = Timeout
26742025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
26752025-03-25T22:33:33.983ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b
26762025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
26772025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26782025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.42\u{b5}s iteration = 7 status = {"enabled":false,"target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b"}
26792025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 11 reason = Timeout
26802025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26812025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 11 reason = Timeout
26822025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
26832025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26842025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 2 reason = Timeout
26852025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 11 reason = Timeout
26862025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
26872025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
26882025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 11 reason = Timeout
26892025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
26902025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
26912025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 11 reason = Timeout
26922025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
26932025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 11 reason = Timeout
26942025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
26952025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 11 reason = Timeout
26962025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26972025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 31 reason = Timeout
26982025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26992025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 6 reason = Timeout
27002025-03-25T22:33:33.983ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
27012025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
27022025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
27032025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 11 reason = Timeout
27042025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
27052025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
27062025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.279504ms iteration = 6 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
27072025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
27082025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
27092025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
27102025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27112025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
27122025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
27132025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
27142025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27152025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27162025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
27172025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
27182025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.345595ms iteration = 6 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
27192025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27202025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
27212025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
27222025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27232025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
27242025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27252025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
27262025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
27272025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27282025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27292025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
27302025-03-25T22:33:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
27312025-03-25T22:33:33.984ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
27322025-03-25T22:33:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
27332025-03-25T22:33:33.984ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:23:33.982164505 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
27342025-03-25T22:33:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 2.783301ms iteration = 6 status = {"expiration":"2025-03-25T22:23:33.982164505Z","pruned":[]}
27352025-03-25T22:33:33.985ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
27362025-03-25T22:33:33.985ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.728408ms iteration = 11 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
27372025-03-25T22:33:33.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1127, tv_nsec: 366685489 } })
27382025-03-25T22:33:33.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
27392025-03-25T22:33:33.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
27402025-03-25T22:33:33.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1127, tv_nsec: 367293550 } })
27412025-03-25T22:33:33.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
27422025-03-25T22:33:33.991ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
27432025-03-25T22:33:33.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
27442025-03-25T22:33:33.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27452025-03-25T22:33:33.995ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27462025-03-25T22:33:33.998ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
27472025-03-25T22:33:33.998ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27482025-03-25T22:33:33.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27492025-03-25T22:33:33.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
27502025-03-25T22:33:33.999ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27512025-03-25T22:33:34.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27522025-03-25T22:33:34.010ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27532025-03-25T22:33:34.012ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 29.526468ms iteration = 6 status = {"errors":[],"found_port_ok":[]}
27542025-03-25T22:33:34.013ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
27552025-03-25T22:33:34.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
27562025-03-25T22:33:34.015ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
27572025-03-25T22:33:34.015ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27582025-03-25T22:33:34.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27592025-03-25T22:33:34.020ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27602025-03-25T22:33:34.023ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
27612025-03-25T22:33:34.023ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
27622025-03-25T22:33:34.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
27632025-03-25T22:33:34.028ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27642025-03-25T22:33:34.032ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
27652025-03-25T22:33:34.032ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
27662025-03-25T22:33:34.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
27672025-03-25T22:33:34.039ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27682025-03-25T22:33:34.044ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
27692025-03-25T22:33:34.046ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27702025-03-25T22:33:34.050ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
27712025-03-25T22:33:34.053ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
27722025-03-25T22:33:34.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
27732025-03-25T22:33:34.058ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27742025-03-25T22:33:34.058ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27752025-03-25T22:33:34.062ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
27762025-03-25T22:33:34.062ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27772025-03-25T22:33:34.062ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
27782025-03-25T22:33:34.062ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
27792025-03-25T22:33:34.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27802025-03-25T22:33:34.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
27812025-03-25T22:33:34.063ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27822025-03-25T22:33:34.065ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27832025-03-25T22:33:34.066ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27842025-03-25T22:33:34.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27852025-03-25T22:33:34.066ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
27862025-03-25T22:33:34.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
27872025-03-25T22:33:34.067ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
27882025-03-25T22:33:34.067ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27892025-03-25T22:33:34.067ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27902025-03-25T22:33:34.069ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
27912025-03-25T22:33:34.069ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
27922025-03-25T22:33:34.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
27932025-03-25T22:33:34.072ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27942025-03-25T22:33:34.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27952025-03-25T22:33:34.072ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27962025-03-25T22:33:34.073ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
27972025-03-25T22:33:34.073ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
27982025-03-25T22:33:34.073ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
27992025-03-25T22:33:34.073ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 91.089352ms iteration = 6 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
28002025-03-25T22:33:34.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
28012025-03-25T22:33:34.075ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 92.202582ms iteration = 11 status = {"errors":[],"garbage_collect_requested":[]}
28022025-03-25T22:33:34.075ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
28032025-03-25T22:33:34.075ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
28042025-03-25T22:33:34.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
28052025-03-25T22:33:34.075ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28062025-03-25T22:33:34.076ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
28072025-03-25T22:33:34.077ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
28082025-03-25T22:33:34.077ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
28092025-03-25T22:33:34.077ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 94.912961ms iteration = 11 status = {"errors":[],"requests_created_ok":[]}
28102025-03-25T22:33:34.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
28112025-03-25T22:33:34.078ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
28122025-03-25T22:33:34.078ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
28132025-03-25T22:33:34.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
28142025-03-25T22:33:34.079ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 96.651353ms iteration = 11 status = {"errors":[],"finish_invoked_ok":[]}
28152025-03-25T22:33:34.079ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28162025-03-25T22:33:34.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28172025-03-25T22:33:34.080ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28182025-03-25T22:33:34.081ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
28192025-03-25T22:33:34.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
28202025-03-25T22:33:34.082ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
28212025-03-25T22:33:34.082ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28222025-03-25T22:33:34.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
28232025-03-25T22:33:34.084ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28242025-03-25T22:33:34.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28252025-03-25T22:33:34.085ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
28262025-03-25T22:33:34.085ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28272025-03-25T22:33:34.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28282025-03-25T22:33:34.086ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
28292025-03-25T22:33:34.086ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
28302025-03-25T22:33:34.086ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
28312025-03-25T22:33:34.088ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
28322025-03-25T22:33:34.089ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
28332025-03-25T22:33:34.090ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
28342025-03-25T22:33:34.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
28352025-03-25T22:33:34.093ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28362025-03-25T22:33:34.094ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28372025-03-25T22:33:34.096ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28382025-03-25T22:33:34.097ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
28392025-03-25T22:33:34.097ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
28402025-03-25T22:33:34.097ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
28412025-03-25T22:33:34.098ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
28422025-03-25T22:33:34.098ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28432025-03-25T22:33:34.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28442025-03-25T22:33:34.098ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
28452025-03-25T22:33:34.099ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
28462025-03-25T22:33:34.099ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
28472025-03-25T22:33:34.099ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
28482025-03-25T22:33:34.099ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
28492025-03-25T22:33:34.099ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
28502025-03-25T22:33:34.100ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
28512025-03-25T22:33:34.100ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28522025-03-25T22:33:34.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28532025-03-25T22:33:34.110ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
28542025-03-25T22:33:34.110ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 128.031805ms iteration = 11 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
28552025-03-25T22:33:34.120ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
28562025-03-25T22:33:34.120ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
28572025-03-25T22:33:34.120ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
28582025-03-25T22:33:34.120ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28592025-03-25T22:33:34.121ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1127, tv_nsec: 501741702 } })
28602025-03-25T22:33:34.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28612025-03-25T22:33:34.121ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
28622025-03-25T22:33:34.122ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
28632025-03-25T22:33:34.122ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
28642025-03-25T22:33:34.122ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 139.768659ms iteration = 31 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
28652025-03-25T22:33:34.124ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28662025-03-25T22:33:34.125ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28672025-03-25T22:33:34.125ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28682025-03-25T22:33:34.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28692025-03-25T22:33:34.127ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 144.645018ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
28702025-03-25T22:33:34.127ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
28712025-03-25T22:33:34.127ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
28722025-03-25T22:33:34.127ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
28732025-03-25T22:33:34.127ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
28742025-03-25T22:33:34.128ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
28752025-03-25T22:33:34.129ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
28762025-03-25T22:33:34.129ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
28772025-03-25T22:33:34.129ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
28782025-03-25T22:33:34.131ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
28792025-03-25T22:33:34.131ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28802025-03-25T22:33:34.131ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28812025-03-25T22:33:34.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28822025-03-25T22:33:34.131ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28832025-03-25T22:33:34.131ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
28842025-03-25T22:33:34.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
28852025-03-25T22:33:34.134ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
28862025-03-25T22:33:34.134ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28872025-03-25T22:33:34.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28882025-03-25T22:33:34.135ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 152.43626ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
28892025-03-25T22:33:34.136ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
28902025-03-25T22:33:34.136ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
28912025-03-25T22:33:34.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
28922025-03-25T22:33:34.139ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
28932025-03-25T22:33:34.139ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
28942025-03-25T22:33:34.139ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
28952025-03-25T22:33:34.139ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
28962025-03-25T22:33:34.139ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
28972025-03-25T22:33:34.139ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa4a9c54-bbe4-4a5a-aac6-44ac747f8799", "content-length": "26", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
28982025-03-25T22:33:34.139ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
28992025-03-25T22:33:34.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
29002025-03-25T22:33:34.142ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
29012025-03-25T22:33:34.146ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29022025-03-25T22:33:34.149ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
29032025-03-25T22:33:34.149ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 166.13445ms iteration = 11 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
29042025-03-25T22:33:34.151ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
29052025-03-25T22:33:34.151ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
29062025-03-25T22:33:34.152ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
29072025-03-25T22:33:34.154ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
29082025-03-25T22:33:34.154ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
29092025-03-25T22:33:34.154ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 170.971178ms iteration = 6 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
29102025-03-25T22:33:34.157ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29112025-03-25T22:33:34.157ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
29122025-03-25T22:33:34.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29132025-03-25T22:33:34.158ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
29142025-03-25T22:33:34.158ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:29:06.674111 UTC dns_group = internal generation = 1
29152025-03-25T22:33:34.158ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 175.821536ms iteration = 6 status = {"generation":1}
29162025-03-25T22:33:34.160ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29172025-03-25T22:33:34.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29182025-03-25T22:33:34.164ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29192025-03-25T22:33:34.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29202025-03-25T22:33:34.165ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 181.966209ms iteration = 31 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
29212025-03-25T22:33:34.166ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29222025-03-25T22:33:34.166ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29232025-03-25T22:33:34.167ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29242025-03-25T22:33:34.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29252025-03-25T22:33:34.170ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29262025-03-25T22:33:34.170ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
29272025-03-25T22:33:34.170ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29282025-03-25T22:33:34.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29292025-03-25T22:33:34.171ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
29302025-03-25T22:33:34.171ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29312025-03-25T22:33:34.171ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29322025-03-25T22:33:34.171ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29332025-03-25T22:33:34.173ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 190.669548ms iteration = 11 status = 0
29342025-03-25T22:33:34.174ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
29352025-03-25T22:33:34.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
29362025-03-25T22:33:34.174ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
29372025-03-25T22:33:34.174ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29382025-03-25T22:33:34.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29392025-03-25T22:33:34.175ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
29402025-03-25T22:33:34.175ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
29412025-03-25T22:33:34.175ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29422025-03-25T22:33:34.176ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29432025-03-25T22:33:34.176ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 193.490409ms iteration = 11 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
29442025-03-25T22:33:34.176ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
29452025-03-25T22:33:34.176ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 0 version = 2
29462025-03-25T22:33:34.176ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
29472025-03-25T22:33:34.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
29482025-03-25T22:33:34.182ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29492025-03-25T22:33:34.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29502025-03-25T22:33:34.183ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29512025-03-25T22:33:34.185ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29522025-03-25T22:33:34.185ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29532025-03-25T22:33:34.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29542025-03-25T22:33:34.186ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29552025-03-25T22:33:34.187ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
29562025-03-25T22:33:34.187ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
29572025-03-25T22:33:34.187ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
29582025-03-25T22:33:34.187ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29592025-03-25T22:33:34.189ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29602025-03-25T22:33:34.190ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
29612025-03-25T22:33:34.190ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
29622025-03-25T22:33:34.190ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 207.697418ms iteration = 11 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
29632025-03-25T22:33:34.190ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
29642025-03-25T22:33:34.190ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29652025-03-25T22:33:34.190ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29662025-03-25T22:33:34.190ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 1 version = 2
29672025-03-25T22:33:34.191ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
29682025-03-25T22:33:34.191ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:29:07.710706 UTC dns_group = external generation = 2
29692025-03-25T22:33:34.191ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 208.850369ms iteration = 6 status = {"generation":2}
29702025-03-25T22:33:34.191ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
29712025-03-25T22:33:34.191ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29722025-03-25T22:33:34.191ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29732025-03-25T22:33:34.194ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29742025-03-25T22:33:34.194ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29752025-03-25T22:33:34.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29762025-03-25T22:33:34.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29772025-03-25T22:33:34.196ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29782025-03-25T22:33:34.197ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29792025-03-25T22:33:34.199ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29802025-03-25T22:33:34.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29812025-03-25T22:33:34.201ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29822025-03-25T22:33:34.201ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29832025-03-25T22:33:34.203ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29842025-03-25T22:33:34.203ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29852025-03-25T22:33:34.211ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:28:31.652727Z, time_modified: 2025-03-25T22:28:31.652727Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }) }
29862025-03-25T22:33:34.211ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 229.103679ms iteration = 6 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]}},"default_endpoint":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
29872025-03-25T22:33:34.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29882025-03-25T22:33:34.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29892025-03-25T22:33:34.247ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29902025-03-25T22:33:34.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29912025-03-25T22:33:34.273ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 290.995338ms iteration = 12 status = {"physical_disks_added":0}
29922025-03-25T22:33:34.275ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
29932025-03-25T22:33:34.275ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 293.089495ms iteration = 6 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
29942025-03-25T22:33:34.290ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
29952025-03-25T22:33:34.290ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
29962025-03-25T22:33:34.290ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1404f6d-86af-49e1-9874-278a26e51f68", "content-length": "26", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
29972025-03-25T22:33:34.290ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
29982025-03-25T22:33:34.292ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
29992025-03-25T22:33:34.384ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
30002025-03-25T22:33:34.384ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
30012025-03-25T22:33:34.384ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "256a5e3f-e925-4fe3-97fe-857000e5ecdb", "content-length": "26", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
30022025-03-25T22:33:34.384ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
30032025-03-25T22:33:34.386ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
30042025-03-25T22:33:34.386ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
30052025-03-25T22:33:34.386ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
30062025-03-25T22:33:34.386ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e29a14f-86c1-4967-b798-9f7b7c3dd35f", "content-length": "372", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
30072025-03-25T22:33:34.386ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30082025-03-25T22:33:34.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30092025-03-25T22:33:34.388ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30102025-03-25T22:33:34.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30112025-03-25T22:33:34.409ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
30122025-03-25T22:33:34.410ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "7394f763-ab64-486e-a7e2-601df98b264f", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
30132025-03-25T22:33:34.410ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
30142025-03-25T22:33:34.411ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "595c8609-4221-415e-893d-9a4a5b5fa99b", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
30152025-03-25T22:33:34.411ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
30162025-03-25T22:33:34.413ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
30172025-03-25T22:33:34.413ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 431.238095ms iteration = 8 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
30182025-03-25T22:33:34.450ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
30192025-03-25T22:33:34.452ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
30202025-03-25T22:33:34.452ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "abfbf100-388c-4789-9a8c-03de2f53c32d", "content-length": "1", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
30212025-03-25T22:33:34.453ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
30222025-03-25T22:33:34.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
30232025-03-25T22:33:34.454ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
30242025-03-25T22:33:34.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
30252025-03-25T22:33:34.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30262025-03-25T22:33:34.455ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30272025-03-25T22:33:34.457ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30282025-03-25T22:33:34.457ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
30292025-03-25T22:33:34.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30302025-03-25T22:33:34.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30312025-03-25T22:33:34.459ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
30322025-03-25T22:33:34.459ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
30332025-03-25T22:33:34.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
30342025-03-25T22:33:34.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
30352025-03-25T22:33:34.461ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30362025-03-25T22:33:34.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30372025-03-25T22:33:34.463ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30382025-03-25T22:33:34.464ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30392025-03-25T22:33:34.464ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 482.567021ms iteration = 11 status = null
30402025-03-25T22:33:34.465ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30412025-03-25T22:33:34.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30422025-03-25T22:33:34.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
30432025-03-25T22:33:34.466ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
30442025-03-25T22:33:34.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
30452025-03-25T22:33:34.467ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
30462025-03-25T22:33:34.467ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
30472025-03-25T22:33:34.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
30482025-03-25T22:33:34.468ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30492025-03-25T22:33:34.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30502025-03-25T22:33:34.469ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
30512025-03-25T22:33:34.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
30522025-03-25T22:33:34.470ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
30532025-03-25T22:33:34.471ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
30542025-03-25T22:33:34.471ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30552025-03-25T22:33:34.471ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "c2dacad3-9c10-4d97-aaba-cadd15bc8338", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
30562025-03-25T22:33:34.471ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30572025-03-25T22:33:34.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30582025-03-25T22:33:34.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30592025-03-25T22:33:34.473ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30602025-03-25T22:33:34.473ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30612025-03-25T22:33:34.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30622025-03-25T22:33:34.473ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30632025-03-25T22:33:34.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30642025-03-25T22:33:34.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30652025-03-25T22:33:34.477ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30662025-03-25T22:33:34.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30672025-03-25T22:33:34.481ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
30682025-03-25T22:33:34.481ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
30692025-03-25T22:33:34.482ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "3547e58c-e7f0-4206-99ff-6aabfc4fb294", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
30702025-03-25T22:33:34.484ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 501.1529ms iteration = 11 status = {}
30712025-03-25T22:33:34.503ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30722025-03-25T22:33:34.503ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 521.497021ms iteration = 11 status = {}
30732025-03-25T22:33:34.505ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
30742025-03-25T22:33:34.505ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
30752025-03-25T22:33:34.505ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
30762025-03-25T22:33:34.506ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30772025-03-25T22:33:34.507ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30782025-03-25T22:33:34.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30792025-03-25T22:33:34.508ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/artifacts-config
30802025-03-25T22:33:34.508ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
30812025-03-25T22:33:34.508ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30822025-03-25T22:33:34.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30832025-03-25T22:33:34.509ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30842025-03-25T22:33:34.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30852025-03-25T22:33:34.511ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30862025-03-25T22:33:34.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30872025-03-25T22:33:34.513ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30882025-03-25T22:33:34.516ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
30892025-03-25T22:33:34.516ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30902025-03-25T22:33:34.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30912025-03-25T22:33:34.518ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30922025-03-25T22:33:34.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30932025-03-25T22:33:34.522ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30942025-03-25T22:33:34.524ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
30952025-03-25T22:33:34.524ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30962025-03-25T22:33:34.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30972025-03-25T22:33:34.527ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30982025-03-25T22:33:34.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30992025-03-25T22:33:34.534ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
31002025-03-25T22:33:34.535ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "e3e331dc-e64a-4775-bdac-ca3178644985", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
31012025-03-25T22:33:34.535ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
31022025-03-25T22:33:34.536ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "95aab517-b18e-4f5d-b45e-5c6269a6e793", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
31032025-03-25T22:33:34.536ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
31042025-03-25T22:33:34.538ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "fd501bf0-5af9-4c65-b72e-e94b23b36a30", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
31052025-03-25T22:33:34.538ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
31062025-03-25T22:33:34.539ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
31072025-03-25T22:33:34.539ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 557.685921ms iteration = 8 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
31082025-03-25T22:33:34.558ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31092025-03-25T22:33:34.561ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
31102025-03-25T22:33:34.561ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31112025-03-25T22:33:34.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31122025-03-25T22:33:34.563ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31132025-03-25T22:33:34.564ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31142025-03-25T22:33:34.566ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31152025-03-25T22:33:34.566ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31162025-03-25T22:33:34.567ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31172025-03-25T22:33:34.567ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31182025-03-25T22:33:34.568ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
31192025-03-25T22:33:34.568ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
31202025-03-25T22:33:34.569ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
31212025-03-25T22:33:34.571ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
31222025-03-25T22:33:34.571ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
31232025-03-25T22:33:34.571ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31242025-03-25T22:33:34.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
31252025-03-25T22:33:34.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31262025-03-25T22:33:34.572ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31272025-03-25T22:33:34.574ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
31282025-03-25T22:33:34.574ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
31292025-03-25T22:33:34.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
31302025-03-25T22:33:34.575ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31312025-03-25T22:33:34.575ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31322025-03-25T22:33:34.576ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
31332025-03-25T22:33:34.576ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
31342025-03-25T22:33:34.577ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31352025-03-25T22:33:34.577ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31362025-03-25T22:33:34.606ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
31372025-03-25T22:33:34.608ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31382025-03-25T22:33:34.609ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31392025-03-25T22:33:34.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31402025-03-25T22:33:34.610ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
31412025-03-25T22:33:34.610ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
31422025-03-25T22:33:34.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
31432025-03-25T22:33:34.611ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31442025-03-25T22:33:34.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31452025-03-25T22:33:34.612ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
31462025-03-25T22:33:34.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
31472025-03-25T22:33:34.613ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31482025-03-25T22:33:34.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31492025-03-25T22:33:34.613ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31502025-03-25T22:33:34.615ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31512025-03-25T22:33:34.615ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
31522025-03-25T22:33:34.615ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
31532025-03-25T22:33:34.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31542025-03-25T22:33:34.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
31552025-03-25T22:33:34.615ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31562025-03-25T22:33:34.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31572025-03-25T22:33:34.616ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31582025-03-25T22:33:34.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31592025-03-25T22:33:34.617ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
31602025-03-25T22:33:34.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
31612025-03-25T22:33:34.618ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
31622025-03-25T22:33:34.618ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31632025-03-25T22:33:34.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31642025-03-25T22:33:34.618ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31652025-03-25T22:33:34.619ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31662025-03-25T22:33:34.620ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "704705b8-0595-467e-948c-05c33e44ad0a", "content-length": "691", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
31672025-03-25T22:33:34.620ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
31682025-03-25T22:33:34.620ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
31692025-03-25T22:33:34.620ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
31702025-03-25T22:33:34.620ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts-config", status: 204, headers: {"x-request-id": "aec74e0a-b4cb-45dd-9773-8b055cd43f33", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
31712025-03-25T22:33:34.620ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
31722025-03-25T22:33:34.621ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "533bf5a0-e83a-4240-b825-4b505a220f4e", "content-length": "26", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
31732025-03-25T22:33:34.621ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 638.580617ms iteration = 11 status = {}
31742025-03-25T22:33:34.621ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
31752025-03-25T22:33:34.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
31762025-03-25T22:33:34.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
31772025-03-25T22:33:34.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31782025-03-25T22:33:34.623ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:673 sled = http://[fd00:1122:3344:101::1]:12345
31792025-03-25T22:33:34.623ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
31802025-03-25T22:33:34.623ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31812025-03-25T22:33:34.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31822025-03-25T22:33:34.625ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31832025-03-25T22:33:34.625ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
31842025-03-25T22:33:34.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
31852025-03-25T22:33:34.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31862025-03-25T22:33:34.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
31872025-03-25T22:33:34.625ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 642.749242ms iteration = 2 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"f7ac1017-36e7-40ec-9172-e4e1545e73c8","time":"2025-03-25T22:33:34.620334724Z"},{"operation":"list","target_sled":"f7ac1017-36e7-40ec-9172-e4e1545e73c8","time":"2025-03-25T22:33:34.623733676Z"}]}
31882025-03-25T22:33:34.626ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31892025-03-25T22:33:34.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31902025-03-25T22:33:34.628ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
31912025-03-25T22:33:34.633ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 645.538013ms iteration = 3 status = {"blueprint_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","inventory_collection_id":"f4145cfd-e21a-49bd-bfc2-974c4a454277","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":8,"num_already_tombstoned":0,"num_inserted":1,"num_not_in_inventory":0,"num_tombstoned":0}}}
31922025-03-25T22:33:34.633ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
31932025-03-25T22:33:34.633ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "04c2f837-7959-46c6-85e1-04c5659db410", "content-length": "134", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
31942025-03-25T22:33:34.633ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
31952025-03-25T22:33:34.633ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
31962025-03-25T22:33:34.633ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
31972025-03-25T22:33:34.633ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
31982025-03-25T22:33:34.633ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
31992025-03-25T22:33:34.633ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32002025-03-25T22:33:34.633ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32012025-03-25T22:33:34.633ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
32022025-03-25T22:33:34.633ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32032025-03-25T22:33:34.633ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
32042025-03-25T22:33:34.633ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
32052025-03-25T22:33:34.633ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32062025-03-25T22:33:34.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32072025-03-25T22:33:34.633ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
32082025-03-25T22:33:34.633ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "871c468e-b968-473e-b351-0f6d9ad92a78", "content-length": "2", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
32092025-03-25T22:33:34.633ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
32102025-03-25T22:33:34.633ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "5062a53f-a700-4512-aaf8-e25763fb85b5", "content-length": "52", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
32112025-03-25T22:33:34.633ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
32122025-03-25T22:33:34.633ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
32132025-03-25T22:33:34.663ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32142025-03-25T22:33:34.670ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
32152025-03-25T22:33:34.670ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
32162025-03-25T22:33:34.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
32172025-03-25T22:33:34.687ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 988adef7-d7d2-40f7-9c58-5a3b26ba2391, kind: Service { id: c8c65163-9425-400d-ac5c-65dfc41d0bf4 }, name: Name("external-dns-c8c65163-9425-400d-ac5c-65dfc41d0bf4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 164])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f88e38f4-26bc-4d44-8958-52780bb3169b, kind: Service { id: 8f1103be-ae21-497e-8fb0-5373f3346d6f }, name: Name("external-dns-8f1103be-ae21-497e-8fb0-5373f3346d6f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 24])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 1658d477-e9f8-4bb8-9566-907b5ca7268e, kind: Service { id: 22035ab1-4462-404d-8fdb-869e194ecaa9 }, name: Name("nexus-22035ab1-4462-404d-8fdb-869e194ecaa9"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 38])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7cb27bfc-265b-486b-9dfd-d94fd10b21da, kind: Service { id: 56ee8e9a-ccae-49c2-ac33-94d5159567dd }, name: Name("nexus-56ee8e9a-ccae-49c2-ac33-94d5159567dd"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 162])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9d699467-8090-45a8-b06c-f08cbb0184d7, kind: Service { id: 94680e6f-9c70-48a1-afd3-ee3593739739 }, name: Name("nexus-94680e6f-9c70-48a1-afd3-ee3593739739"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 189, 114])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
32182025-03-25T22:33:34.687ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
32192025-03-25T22:33:34.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
32202025-03-25T22:33:34.691ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32212025-03-25T22:33:34.695ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
32222025-03-25T22:33:34.695ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
32232025-03-25T22:33:34.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
32242025-03-25T22:33:34.697ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
32252025-03-25T22:33:34.700ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 988adef7-d7d2-40f7-9c58-5a3b26ba2391, kind: Service { id: c8c65163-9425-400d-ac5c-65dfc41d0bf4 }, name: Name("external-dns-c8c65163-9425-400d-ac5c-65dfc41d0bf4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 164])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f88e38f4-26bc-4d44-8958-52780bb3169b, kind: Service { id: 8f1103be-ae21-497e-8fb0-5373f3346d6f }, name: Name("external-dns-8f1103be-ae21-497e-8fb0-5373f3346d6f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 24])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 1658d477-e9f8-4bb8-9566-907b5ca7268e, kind: Service { id: 22035ab1-4462-404d-8fdb-869e194ecaa9 }, name: Name("nexus-22035ab1-4462-404d-8fdb-869e194ecaa9"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 38])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7cb27bfc-265b-486b-9dfd-d94fd10b21da, kind: Service { id: 56ee8e9a-ccae-49c2-ac33-94d5159567dd }, name: Name("nexus-56ee8e9a-ccae-49c2-ac33-94d5159567dd"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 162])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9d699467-8090-45a8-b06c-f08cbb0184d7, kind: Service { id: 94680e6f-9c70-48a1-afd3-ee3593739739 }, name: Name("nexus-94680e6f-9c70-48a1-afd3-ee3593739739"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 189, 114])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
32262025-03-25T22:33:34.700ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
32272025-03-25T22:33:34.768ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "630d0075-937b-41bc-8886-9929712b1537", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
32282025-03-25T22:33:34.768ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
32292025-03-25T22:33:34.768ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
32302025-03-25T22:33:34.769ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "03b8295e-234d-4ce7-971a-d6d5502e33c0", "content-length": "682", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
32312025-03-25T22:33:34.769ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32322025-03-25T22:33:34.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32332025-03-25T22:33:34.773ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32342025-03-25T22:33:34.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32352025-03-25T22:33:34.777ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32362025-03-25T22:33:34.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32372025-03-25T22:33:34.779ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32382025-03-25T22:33:34.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32392025-03-25T22:33:34.786ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 803.749568ms iteration = 11 status = {}
32402025-03-25T22:33:34.851ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: f7ac1017-36e7-40ec-9172-e4e1545e73c8, time_created: 2025-03-25T22:29:06.944737Z, time_modified: 2025-03-25T22:29:06.944737Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
32412025-03-25T22:33:34.851ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
32422025-03-25T22:33:34.855ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
32432025-03-25T22:33:34.855ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
32442025-03-25T22:33:34.858ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
32452025-03-25T22:33:34.858ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
32462025-03-25T22:33:34.859ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32472025-03-25T22:33:34.864ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
32482025-03-25T22:33:34.901ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
32492025-03-25T22:33:34.977ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "411eeb89-d368-403f-941f-cf6dbbec3032", "date": "Tue, 25 Mar 2025 22:33:34 GMT"} })
32502025-03-25T22:33:34.978ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
32512025-03-25T22:33:34.978ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 995.21647ms
32522025-03-25T22:33:34.978ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 995.238531ms iteration = 2 status = {}
32532025-03-25T22:33:36.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 271 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = f1cd2179-993d-4914-9f6e-63268a08a46d response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
32542025-03-25T22:33:43.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 32 reason = Timeout
32552025-03-25T22:33:43.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
32562025-03-25T22:33:43.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32572025-03-25T22:33:43.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32582025-03-25T22:33:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32592025-03-25T22:33:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32602025-03-25T22:33:43.984ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32612025-03-25T22:33:43.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32622025-03-25T22:33:43.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32632025-03-25T22:33:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
32642025-03-25T22:33:43.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32652025-03-25T22:33:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32662025-03-25T22:33:43.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32672025-03-25T22:33:43.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32682025-03-25T22:33:43.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32692025-03-25T22:33:43.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
32702025-03-25T22:33:43.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32712025-03-25T22:33:43.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32722025-03-25T22:33:44.003ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
32732025-03-25T22:33:44.003ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.090676ms iteration = 32 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
32742025-03-25T22:33:44.015ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.520223ms iteration = 32 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
32752025-03-25T22:33:46.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 6a5a048f-dc95-4fa5-8581-1dcf1ebc11bc roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
32762025-03-25T22:33:46.079ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 6a5a048f-dc95-4fa5-8581-1dcf1ebc11bc resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
32772025-03-25T22:33:46.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3005 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 6a5a048f-dc95-4fa5-8581-1dcf1ebc11bc response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
32782025-03-25T22:33:46.082ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = c3a12fe9-6584-47e8-a9fa-0fd56484c093 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
32792025-03-25T22:33:46.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = c3a12fe9-6584-47e8-a9fa-0fd56484c093 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
32802025-03-25T22:33:46.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2281 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = c3a12fe9-6584-47e8-a9fa-0fd56484c093 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
32812025-03-25T22:33:46.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 241 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 15f79f30-1bf0-49ef-bbca-5a8a9455fb46 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
32822025-03-25T22:33:53.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
32832025-03-25T22:33:53.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 33 reason = Timeout
32842025-03-25T22:33:53.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32852025-03-25T22:33:53.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32862025-03-25T22:33:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32872025-03-25T22:33:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32882025-03-25T22:33:53.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32892025-03-25T22:33:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
32902025-03-25T22:33:53.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32912025-03-25T22:33:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32922025-03-25T22:33:53.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32932025-03-25T22:33:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32942025-03-25T22:33:53.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32952025-03-25T22:33:53.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
32962025-03-25T22:33:53.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32972025-03-25T22:33:53.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32982025-03-25T22:33:54.011ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
32992025-03-25T22:33:54.011ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.604143ms iteration = 33 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
33002025-03-25T22:33:54.020ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33012025-03-25T22:33:54.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33022025-03-25T22:33:54.051ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 68.140163ms iteration = 33 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
33032025-03-25T22:33:56.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 290 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = b00ab12c-c257-4d37-9263-730394c7a132 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
33042025-03-25T22:34:03.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
33052025-03-25T22:34:03.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
33062025-03-25T22:34:03.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
33072025-03-25T22:34:03.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
33082025-03-25T22:34:03.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
33092025-03-25T22:34:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
33102025-03-25T22:34:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
33112025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 12 reason = Timeout
33122025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 12 reason = Timeout
33132025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 12 reason = Timeout
33142025-03-25T22:34:03.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
33152025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
33162025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
33172025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
33182025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33192025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 12 reason = Timeout
33202025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
33212025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
33222025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 12 reason = Timeout
33232025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
33242025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
33252025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
33262025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 34 reason = Timeout
33272025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33282025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33292025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 12 reason = Timeout
33302025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 12 reason = Timeout
33312025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
33322025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 12 reason = Timeout
33332025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33342025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
33352025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 12 reason = Timeout
33362025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
33372025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 13 reason = Timeout
33382025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33392025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 12 reason = Timeout
33402025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
33412025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 12 reason = Timeout
33422025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
33432025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
33442025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
33452025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
33462025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33472025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
33482025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
33492025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
33502025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33512025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33522025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
33532025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33542025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33552025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
33562025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
33572025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
33582025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33592025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
33602025-03-25T22:34:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33612025-03-25T22:34:03.984ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1157, tv_nsec: 365095460 } })
33622025-03-25T22:34:03.984ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
33632025-03-25T22:34:03.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1157, tv_nsec: 365753802 } })
33642025-03-25T22:34:03.985ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
33652025-03-25T22:34:03.986ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.088765ms iteration = 12 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
33662025-03-25T22:34:03.990ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33672025-03-25T22:34:03.990ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33682025-03-25T22:34:03.993ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
33692025-03-25T22:34:03.993ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33702025-03-25T22:34:03.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33712025-03-25T22:34:03.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
33722025-03-25T22:34:03.994ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
33732025-03-25T22:34:03.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
33742025-03-25T22:34:03.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33752025-03-25T22:34:03.998ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33762025-03-25T22:34:04.000ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
33772025-03-25T22:34:04.000ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33782025-03-25T22:34:04.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33792025-03-25T22:34:04.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
33802025-03-25T22:34:04.001ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33812025-03-25T22:34:04.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33822025-03-25T22:34:04.005ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33832025-03-25T22:34:04.009ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
33842025-03-25T22:34:04.009ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33852025-03-25T22:34:04.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33862025-03-25T22:34:04.013ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33872025-03-25T22:34:04.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33882025-03-25T22:34:04.015ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33892025-03-25T22:34:04.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33902025-03-25T22:34:04.015ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33912025-03-25T22:34:04.019ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
33922025-03-25T22:34:04.019ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33932025-03-25T22:34:04.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33942025-03-25T22:34:04.019ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33952025-03-25T22:34:04.022ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1157, tv_nsec: 402903480 } })
33962025-03-25T22:34:04.022ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
33972025-03-25T22:34:04.022ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
33982025-03-25T22:34:04.022ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
33992025-03-25T22:34:04.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34002025-03-25T22:34:04.039ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
34012025-03-25T22:34:04.039ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 56.633533ms iteration = 34 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
34022025-03-25T22:34:04.052ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
34032025-03-25T22:34:04.053ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
34042025-03-25T22:34:04.053ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 69.845924ms iteration = 12 status = {"errors":[],"garbage_collect_requested":[]}
34052025-03-25T22:34:04.055ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
34062025-03-25T22:34:04.055ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
34072025-03-25T22:34:04.055ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
34082025-03-25T22:34:04.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
34092025-03-25T22:34:04.056ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34102025-03-25T22:34:04.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34112025-03-25T22:34:04.059ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34122025-03-25T22:34:04.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34132025-03-25T22:34:04.068ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
34142025-03-25T22:34:04.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
34152025-03-25T22:34:04.069ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
34162025-03-25T22:34:04.069ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
34172025-03-25T22:34:04.070ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "39050b22-3184-4164-8ce6-6bdb14249bd1", "content-length": "26", "date": "Tue, 25 Mar 2025 22:34:04 GMT"} })
34182025-03-25T22:34:04.070ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
34192025-03-25T22:34:04.071ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
34202025-03-25T22:34:04.074ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 91.109861ms iteration = 12 status = {"errors":[],"requests_created_ok":[]}
34212025-03-25T22:34:04.074ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 91.717203ms iteration = 12 status = {"errors":[],"finish_invoked_ok":[]}
34222025-03-25T22:34:04.076ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
34232025-03-25T22:34:04.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
34242025-03-25T22:34:04.077ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34252025-03-25T22:34:04.079ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34262025-03-25T22:34:04.079ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34272025-03-25T22:34:04.080ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34282025-03-25T22:34:04.080ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
34292025-03-25T22:34:04.080ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 97.59842ms iteration = 12 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
34302025-03-25T22:34:04.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34312025-03-25T22:34:04.081ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
34322025-03-25T22:34:04.081ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34332025-03-25T22:34:04.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34342025-03-25T22:34:04.082ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
34352025-03-25T22:34:04.082ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
34362025-03-25T22:34:04.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
34372025-03-25T22:34:04.084ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
34382025-03-25T22:34:04.084ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34392025-03-25T22:34:04.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34402025-03-25T22:34:04.087ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
34412025-03-25T22:34:04.087ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
34422025-03-25T22:34:04.088ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f1220b8-7e58-4a48-b62b-c1bde31a48f9", "content-length": "26", "date": "Tue, 25 Mar 2025 22:34:04 GMT"} })
34432025-03-25T22:34:04.088ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34442025-03-25T22:34:04.088ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
34452025-03-25T22:34:04.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34462025-03-25T22:34:04.090ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
34472025-03-25T22:34:04.109ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
34482025-03-25T22:34:04.109ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
34492025-03-25T22:34:04.114ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 130.805795ms iteration = 34 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
34502025-03-25T22:34:04.117ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
34512025-03-25T22:34:04.117ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
34522025-03-25T22:34:04.118ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5378af1-bb1f-4792-8085-feb1277c36ea", "content-length": "26", "date": "Tue, 25 Mar 2025 22:34:04 GMT"} })
34532025-03-25T22:34:04.118ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
34542025-03-25T22:34:04.120ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
34552025-03-25T22:34:04.120ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 137.716752ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
34562025-03-25T22:34:04.124ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
34572025-03-25T22:34:04.124ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
34582025-03-25T22:34:04.124ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
34592025-03-25T22:34:04.125ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
34602025-03-25T22:34:04.128ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 145.476453ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
34612025-03-25T22:34:04.129ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34622025-03-25T22:34:04.129ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34632025-03-25T22:34:04.131ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34642025-03-25T22:34:04.132ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34652025-03-25T22:34:04.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34662025-03-25T22:34:04.134ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34672025-03-25T22:34:04.136ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
34682025-03-25T22:34:04.136ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34692025-03-25T22:34:04.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34702025-03-25T22:34:04.137ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
34712025-03-25T22:34:04.137ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34722025-03-25T22:34:04.137ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34732025-03-25T22:34:04.137ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 154.04994ms iteration = 13 status = {"physical_disks_added":0}
34742025-03-25T22:34:04.139ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 155.966185ms iteration = 12 status = 0
34752025-03-25T22:34:04.148ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 165.01766ms iteration = 12 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
34762025-03-25T22:34:04.161ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
34772025-03-25T22:34:04.161ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
34782025-03-25T22:34:04.161ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 178.851492ms iteration = 12 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
34792025-03-25T22:34:04.169ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
34802025-03-25T22:34:04.169ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 186.257427ms iteration = 12 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
34812025-03-25T22:34:04.171ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 189.624709ms iteration = 12 status = {}
34822025-03-25T22:34:04.185ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34832025-03-25T22:34:04.186ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34842025-03-25T22:34:04.187ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34852025-03-25T22:34:04.189ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
34862025-03-25T22:34:04.189ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34872025-03-25T22:34:04.189ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34882025-03-25T22:34:04.191ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34892025-03-25T22:34:04.193ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
34902025-03-25T22:34:04.193ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34912025-03-25T22:34:04.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34922025-03-25T22:34:04.262ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
34932025-03-25T22:34:04.262ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
34942025-03-25T22:34:04.262ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "115e7768-6be1-4c54-bab1-2c05b0a224b3", "content-length": "372", "date": "Tue, 25 Mar 2025 22:34:04 GMT"} })
34952025-03-25T22:34:04.263ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34962025-03-25T22:34:04.263ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34972025-03-25T22:34:04.267ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
34982025-03-25T22:34:04.268ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e500ff6-3953-4e7d-9da7-1104b34509b9", "content-length": "1", "date": "Tue, 25 Mar 2025 22:34:03 GMT"} })
34992025-03-25T22:34:04.270ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
35002025-03-25T22:34:04.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
35012025-03-25T22:34:04.272ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 290.05375ms iteration = 12 status = null
35022025-03-25T22:34:04.274ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
35032025-03-25T22:34:04.274ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
35042025-03-25T22:34:04.275ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "dba4f501-97e7-43a2-a1a3-ee77eea0c163", "date": "Tue, 25 Mar 2025 22:34:04 GMT"} })
35052025-03-25T22:34:04.275ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35062025-03-25T22:34:04.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35072025-03-25T22:34:04.277ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35082025-03-25T22:34:04.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35092025-03-25T22:34:04.279ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35102025-03-25T22:34:04.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35112025-03-25T22:34:04.283ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
35122025-03-25T22:34:04.283ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
35132025-03-25T22:34:04.284ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "bac39b7f-c8be-4d4b-b3e5-7145595ba0a1", "date": "Tue, 25 Mar 2025 22:34:04 GMT"} })
35142025-03-25T22:34:04.286ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 302.930395ms iteration = 12 status = {}
35152025-03-25T22:34:04.363ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35162025-03-25T22:34:04.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35172025-03-25T22:34:04.366ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
35182025-03-25T22:34:04.366ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35192025-03-25T22:34:04.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35202025-03-25T22:34:04.383ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
35212025-03-25T22:34:04.384ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf90986f-88d3-4a95-b40c-79dbaff9806b", "content-length": "134", "date": "Tue, 25 Mar 2025 22:34:03 GMT"} })
35222025-03-25T22:34:04.384ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
35232025-03-25T22:34:04.384ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
35242025-03-25T22:34:04.384ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
35252025-03-25T22:34:04.384ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
35262025-03-25T22:34:04.384ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
35272025-03-25T22:34:04.384ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
35282025-03-25T22:34:04.384ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
35292025-03-25T22:34:04.384ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
35302025-03-25T22:34:04.385ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
35312025-03-25T22:34:04.385ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
35322025-03-25T22:34:04.385ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
35332025-03-25T22:34:04.385ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35342025-03-25T22:34:04.385ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
35352025-03-25T22:34:04.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35362025-03-25T22:34:04.389ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
35372025-03-25T22:34:04.389ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c1e3268-b59e-4ed3-88fa-ab5ce03ce183", "content-length": "2", "date": "Tue, 25 Mar 2025 22:34:03 GMT"} })
35382025-03-25T22:34:04.389ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
35392025-03-25T22:34:04.389ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "80105eca-f504-4e8d-8aa7-480a685a5c36", "content-length": "52", "date": "Tue, 25 Mar 2025 22:34:03 GMT"} })
35402025-03-25T22:34:04.390ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
35412025-03-25T22:34:04.390ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
35422025-03-25T22:34:04.451ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d0ee60d-b8e0-467c-83d0-baa29dc75f64", "content-length": "691", "date": "Tue, 25 Mar 2025 22:34:04 GMT"} })
35432025-03-25T22:34:04.451ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
35442025-03-25T22:34:04.451ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
35452025-03-25T22:34:04.451ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
35462025-03-25T22:34:04.453ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 469.795268ms iteration = 12 status = {}
35472025-03-25T22:34:04.557ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "d6ef3ce3-e07a-452b-9384-4a36fd7b4000", "date": "Tue, 25 Mar 2025 22:34:04 GMT"} })
35482025-03-25T22:34:04.557ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
35492025-03-25T22:34:04.557ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
35502025-03-25T22:34:04.557ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dabcf4f-def0-468c-97bc-3fda9823127d", "content-length": "682", "date": "Tue, 25 Mar 2025 22:34:04 GMT"} })
35512025-03-25T22:34:04.557ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35522025-03-25T22:34:04.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35532025-03-25T22:34:04.561ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35542025-03-25T22:34:04.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35552025-03-25T22:34:04.565ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35562025-03-25T22:34:04.565ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35572025-03-25T22:34:04.567ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35582025-03-25T22:34:04.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35592025-03-25T22:34:04.575ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 592.011528ms iteration = 12 status = {}
35602025-03-25T22:34:06.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 330 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = c32d764e-d5a9-4c65-af17-55b05c5c61aa response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
35612025-03-25T22:34:13.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
35622025-03-25T22:34:13.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 35 reason = Timeout
35632025-03-25T22:34:13.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35642025-03-25T22:34:13.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
35652025-03-25T22:34:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35662025-03-25T22:34:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35672025-03-25T22:34:13.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
35682025-03-25T22:34:13.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35692025-03-25T22:34:13.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35702025-03-25T22:34:13.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
35712025-03-25T22:34:13.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35722025-03-25T22:34:13.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35732025-03-25T22:34:13.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35742025-03-25T22:34:13.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35752025-03-25T22:34:13.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35762025-03-25T22:34:13.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
35772025-03-25T22:34:13.997ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35782025-03-25T22:34:13.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35792025-03-25T22:34:14.005ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
35802025-03-25T22:34:14.005ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.390158ms iteration = 35 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
35812025-03-25T22:34:14.018ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 35.258823ms iteration = 35 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
35822025-03-25T22:34:16.077ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 681d0ed8-24b8-4ee3-8c2e-2f67ebe71e55 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
35832025-03-25T22:34:16.078ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 681d0ed8-24b8-4ee3-8c2e-2f67ebe71e55 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
35842025-03-25T22:34:16.080ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3129 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 681d0ed8-24b8-4ee3-8c2e-2f67ebe71e55 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
35852025-03-25T22:34:16.081ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = d2d91980-8738-4693-bd73-6dd14234a5fa roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
35862025-03-25T22:34:16.081ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = d2d91980-8738-4693-bd73-6dd14234a5fa resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
35872025-03-25T22:34:16.083ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2735 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = d2d91980-8738-4693-bd73-6dd14234a5fa response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
35882025-03-25T22:34:16.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 311 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = b55a6e5b-2922-4d3b-ba89-91404e6ef9e6 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
35892025-03-25T22:34:23.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
35902025-03-25T22:34:23.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 36 reason = Timeout
35912025-03-25T22:34:23.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
35922025-03-25T22:34:23.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35932025-03-25T22:34:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35942025-03-25T22:34:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35952025-03-25T22:34:23.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35962025-03-25T22:34:23.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
35972025-03-25T22:34:23.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35982025-03-25T22:34:23.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.583506ms iteration = 36 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
35992025-03-25T22:34:23.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
36002025-03-25T22:34:23.987ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36012025-03-25T22:34:23.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36022025-03-25T22:34:23.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36032025-03-25T22:34:23.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36042025-03-25T22:34:23.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36052025-03-25T22:34:23.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
36062025-03-25T22:34:23.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36072025-03-25T22:34:23.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36082025-03-25T22:34:23.999ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
36092025-03-25T22:34:23.999ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.241966ms iteration = 36 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
36102025-03-25T22:34:26.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 262 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = e83df744-04aa-470c-a461-625528116dcd response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
36112025-03-25T22:34:31.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 4eac39ef-4cb1-4082-b768-9055ac60d0c2 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
36122025-03-25T22:34:31.078ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 4eac39ef-4cb1-4082-b768-9055ac60d0c2 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
36132025-03-25T22:34:31.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3077 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 4eac39ef-4cb1-4082-b768-9055ac60d0c2 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
36142025-03-25T22:34:31.081ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 51f6abb4-cea2-4954-bd90-c2c734d1163d roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
36152025-03-25T22:34:31.081ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 51f6abb4-cea2-4954-bd90-c2c734d1163d resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
36162025-03-25T22:34:31.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2471 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 51f6abb4-cea2-4954-bd90-c2c734d1163d response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
36172025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
36182025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
36192025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 7 reason = Timeout
36202025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
36212025-03-25T22:34:33.982ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:24:33.982422969 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
36222025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
36232025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
36242025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
36252025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
36262025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
36272025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
36282025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
36292025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
36302025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
36312025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
36322025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
36332025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
36342025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
36352025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
36362025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
36372025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
36382025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
36392025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 13 reason = Timeout
36402025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 13 reason = Timeout
36412025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
36422025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
36432025-03-25T22:34:33.982ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
36442025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
36452025-03-25T22:34:33.983ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b
36462025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
36472025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 14 reason = Timeout
36482025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.601\u{b5}s iteration = 8 status = {"enabled":false,"target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b"}
36492025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
36502025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
36512025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 13 reason = Timeout
36522025-03-25T22:34:33.983ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("09fed906-08a5-4e51-9e85-fba7211b9594.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4ba21a7b-c053-4a77-807f-34395131fce2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1187, tv_nsec: 363531451 } })
36532025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 4 reason = Timeout
36542025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36552025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
36562025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
36572025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
36582025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
36592025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 13 reason = Timeout
36602025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36612025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36622025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 13 reason = Timeout
36632025-03-25T22:34:33.983ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8f1103be-ae21-497e-8fb0-5373f3346d6f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1187, tv_nsec: 363558022 } })
36642025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
36652025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
36662025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 13 reason = Timeout
36672025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
36682025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 7 reason = Timeout
36692025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
36702025-03-25T22:34:33.983ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
36712025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 13 reason = Timeout
36722025-03-25T22:34:33.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
36732025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
36742025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 7 reason = Timeout
36752025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
36762025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
36772025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 13 reason = Timeout
36782025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
36792025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36802025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
36812025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 7 reason = Timeout
36822025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
36832025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 13 reason = Timeout
36842025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
36852025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 7 reason = Timeout
36862025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
36872025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 37 reason = Timeout
36882025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 13 reason = Timeout
36892025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36902025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 13 reason = Timeout
36912025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
36922025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36932025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
36942025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36952025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
36962025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
36972025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36982025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
36992025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
37002025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
37012025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37022025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37032025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37042025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
37052025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
37062025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
37072025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
37082025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
37092025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
37102025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.15545ms iteration = 7 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
37112025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
37122025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.259783ms iteration = 7 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
37132025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
37142025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
37152025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
37162025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
37172025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
37182025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37192025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37202025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37212025-03-25T22:34:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37222025-03-25T22:34:33.984ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37232025-03-25T22:34:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37242025-03-25T22:34:33.984ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37252025-03-25T22:34:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37262025-03-25T22:34:33.984ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
37272025-03-25T22:34:33.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37282025-03-25T22:34:33.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1187, tv_nsec: 365808373 } })
37292025-03-25T22:34:33.985ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37302025-03-25T22:34:33.985ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
37312025-03-25T22:34:33.985ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:24:33.982422969 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
37322025-03-25T22:34:33.985ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.28037ms iteration = 7 status = {"expiration":"2025-03-25T22:24:33.982422969Z","pruned":[]}
37332025-03-25T22:34:33.985ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
37342025-03-25T22:34:33.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
37352025-03-25T22:34:33.986ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.100785ms iteration = 13 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
37362025-03-25T22:34:33.987ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37372025-03-25T22:34:33.987ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
37382025-03-25T22:34:33.987ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
37392025-03-25T22:34:33.987ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1187, tv_nsec: 367948862 } })
37402025-03-25T22:34:33.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
37412025-03-25T22:34:33.987ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
37422025-03-25T22:34:33.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37432025-03-25T22:34:33.989ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 6.56764ms iteration = 13 status = {"errors":[],"garbage_collect_requested":[]}
37442025-03-25T22:34:33.990ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37452025-03-25T22:34:33.990ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
37462025-03-25T22:34:33.990ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
37472025-03-25T22:34:33.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
37482025-03-25T22:34:33.992ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
37492025-03-25T22:34:33.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37502025-03-25T22:34:33.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37512025-03-25T22:34:33.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
37522025-03-25T22:34:33.993ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37532025-03-25T22:34:33.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37542025-03-25T22:34:33.994ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37552025-03-25T22:34:33.994ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 11.801715ms iteration = 13 status = {"errors":[],"finish_invoked_ok":[]}
37562025-03-25T22:34:33.995ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37572025-03-25T22:34:33.995ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
37582025-03-25T22:34:33.996ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37592025-03-25T22:34:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
37602025-03-25T22:34:33.997ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37612025-03-25T22:34:33.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
37622025-03-25T22:34:33.998ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37632025-03-25T22:34:33.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
37642025-03-25T22:34:33.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
37652025-03-25T22:34:33.999ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37662025-03-25T22:34:34.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37672025-03-25T22:34:34.000ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
37682025-03-25T22:34:34.000ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
37692025-03-25T22:34:34.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
37702025-03-25T22:34:34.000ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
37712025-03-25T22:34:34.000ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37722025-03-25T22:34:34.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37732025-03-25T22:34:34.018ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 35.149381ms iteration = 7 status = {"errors":[],"found_port_ok":[]}
37742025-03-25T22:34:34.019ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37752025-03-25T22:34:34.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
37762025-03-25T22:34:34.024ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
37772025-03-25T22:34:34.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
37782025-03-25T22:34:34.025ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37792025-03-25T22:34:34.028ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37802025-03-25T22:34:34.028ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
37812025-03-25T22:34:34.029ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37822025-03-25T22:34:34.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37832025-03-25T22:34:34.035ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
37842025-03-25T22:34:34.037ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
37852025-03-25T22:34:34.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
37862025-03-25T22:34:34.039ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37872025-03-25T22:34:34.039ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37882025-03-25T22:34:34.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37892025-03-25T22:34:34.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37902025-03-25T22:34:34.040ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37912025-03-25T22:34:34.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
37922025-03-25T22:34:34.042ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37932025-03-25T22:34:34.046ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
37942025-03-25T22:34:34.046ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37952025-03-25T22:34:34.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37962025-03-25T22:34:34.066ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37972025-03-25T22:34:34.069ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
37982025-03-25T22:34:34.069ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37992025-03-25T22:34:34.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38002025-03-25T22:34:34.074ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 91.899586ms iteration = 13 status = {"errors":[],"requests_created_ok":[]}
38012025-03-25T22:34:34.077ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
38022025-03-25T22:34:34.077ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
38032025-03-25T22:34:34.077ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 94.791089ms iteration = 7 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
38042025-03-25T22:34:34.080ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
38052025-03-25T22:34:34.080ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1187, tv_nsec: 461489458 } })
38062025-03-25T22:34:34.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
38072025-03-25T22:34:34.081ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
38082025-03-25T22:34:34.082ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38092025-03-25T22:34:34.084ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
38102025-03-25T22:34:34.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
38112025-03-25T22:34:34.085ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
38122025-03-25T22:34:34.085ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
38132025-03-25T22:34:34.085ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
38142025-03-25T22:34:34.086ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
38152025-03-25T22:34:34.086ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
38162025-03-25T22:34:34.086ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
38172025-03-25T22:34:34.086ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
38182025-03-25T22:34:34.087ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
38192025-03-25T22:34:34.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
38202025-03-25T22:34:34.089ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38212025-03-25T22:34:34.089ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38222025-03-25T22:34:34.093ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
38232025-03-25T22:34:34.093ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
38242025-03-25T22:34:34.093ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38252025-03-25T22:34:34.093ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
38262025-03-25T22:34:34.094ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
38272025-03-25T22:34:34.094ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38282025-03-25T22:34:34.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38292025-03-25T22:34:34.097ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 114.61692ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
38302025-03-25T22:34:34.097ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
38312025-03-25T22:34:34.097ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
38322025-03-25T22:34:34.097ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
38332025-03-25T22:34:34.097ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
38342025-03-25T22:34:34.097ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
38352025-03-25T22:34:34.097ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
38362025-03-25T22:34:34.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
38372025-03-25T22:34:34.098ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38382025-03-25T22:34:34.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
38392025-03-25T22:34:34.101ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
38402025-03-25T22:34:34.101ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
38412025-03-25T22:34:34.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
38422025-03-25T22:34:34.104ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38432025-03-25T22:34:34.107ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
38442025-03-25T22:34:34.107ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
38452025-03-25T22:34:34.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
38462025-03-25T22:34:34.110ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38472025-03-25T22:34:34.112ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38482025-03-25T22:34:34.114ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
38492025-03-25T22:34:34.114ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38502025-03-25T22:34:34.114ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38512025-03-25T22:34:34.117ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
38522025-03-25T22:34:34.117ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
38532025-03-25T22:34:34.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
38542025-03-25T22:34:34.120ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38552025-03-25T22:34:34.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38562025-03-25T22:34:34.121ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
38572025-03-25T22:34:34.121ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
38582025-03-25T22:34:34.121ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "accb1d5f-aecc-4b2e-861b-536849344bbc", "content-length": "26", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} })
38592025-03-25T22:34:34.121ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
38602025-03-25T22:34:34.123ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
38612025-03-25T22:34:34.128ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38622025-03-25T22:34:34.129ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 146.084104ms iteration = 4 status = {"nsuccess":5}
38632025-03-25T22:34:34.129ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38642025-03-25T22:34:34.129ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
38652025-03-25T22:34:34.129ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
38662025-03-25T22:34:34.129ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 146.533383ms iteration = 7 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
38672025-03-25T22:34:34.132ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38682025-03-25T22:34:34.134ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
38692025-03-25T22:34:34.136ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
38702025-03-25T22:34:34.136ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38712025-03-25T22:34:34.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38722025-03-25T22:34:34.138ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 155.863522ms iteration = 13 status = 0
38732025-03-25T22:34:34.144ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
38742025-03-25T22:34:34.144ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 161.74231ms iteration = 13 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
38752025-03-25T22:34:34.147ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38762025-03-25T22:34:34.152ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
38772025-03-25T22:34:34.152ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38782025-03-25T22:34:34.152ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38792025-03-25T22:34:34.159ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 176.00828ms iteration = 37 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
38802025-03-25T22:34:34.162ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
38812025-03-25T22:34:34.162ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
38822025-03-25T22:34:34.163ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 180.40064ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
38832025-03-25T22:34:34.164ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
38842025-03-25T22:34:34.164ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
38852025-03-25T22:34:34.164ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
38862025-03-25T22:34:34.164ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
38872025-03-25T22:34:34.164ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
38882025-03-25T22:34:34.164ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 181.351798ms iteration = 37 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
38892025-03-25T22:34:34.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
38902025-03-25T22:34:34.166ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
38912025-03-25T22:34:34.166ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38922025-03-25T22:34:34.166ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
38932025-03-25T22:34:34.167ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
38942025-03-25T22:34:34.168ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
38952025-03-25T22:34:34.168ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
38962025-03-25T22:34:34.168ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
38972025-03-25T22:34:34.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
38982025-03-25T22:34:34.169ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
38992025-03-25T22:34:34.169ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
39002025-03-25T22:34:34.170ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
39012025-03-25T22:34:34.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
39022025-03-25T22:34:34.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
39032025-03-25T22:34:34.171ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39042025-03-25T22:34:34.173ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
39052025-03-25T22:34:34.173ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
39062025-03-25T22:34:34.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
39072025-03-25T22:34:34.173ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
39082025-03-25T22:34:34.173ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 190.898732ms iteration = 13 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
39092025-03-25T22:34:34.178ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
39102025-03-25T22:34:34.178ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
39112025-03-25T22:34:34.178ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
39122025-03-25T22:34:34.178ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
39132025-03-25T22:34:34.180ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39142025-03-25T22:34:34.183ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
39152025-03-25T22:34:34.183ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
39162025-03-25T22:34:34.184ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
39172025-03-25T22:34:34.187ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
39182025-03-25T22:34:34.187ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
39192025-03-25T22:34:34.187ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:29:06.674111 UTC dns_group = internal generation = 1
39202025-03-25T22:34:34.187ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 204.603401ms iteration = 7 status = {"generation":1}
39212025-03-25T22:34:34.202ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39222025-03-25T22:34:34.203ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39232025-03-25T22:34:34.207ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
39242025-03-25T22:34:34.207ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
39252025-03-25T22:34:34.207ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c342636-5271-4026-9217-f7ac5db14943", "content-length": "26", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} })
39262025-03-25T22:34:34.207ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
39272025-03-25T22:34:34.209ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
39282025-03-25T22:34:34.216ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 233.705813ms iteration = 13 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
39292025-03-25T22:34:34.218ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
39302025-03-25T22:34:34.218ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 0 version = 2
39312025-03-25T22:34:34.218ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
39322025-03-25T22:34:34.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
39332025-03-25T22:34:34.218ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
39342025-03-25T22:34:34.218ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
39352025-03-25T22:34:34.218ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 235.612517ms iteration = 13 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
39362025-03-25T22:34:34.220ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39372025-03-25T22:34:34.224ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
39382025-03-25T22:34:34.224ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
39392025-03-25T22:34:34.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
39402025-03-25T22:34:34.227ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 1 version = 2
39412025-03-25T22:34:34.227ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
39422025-03-25T22:34:34.227ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:29:07.710706 UTC dns_group = external generation = 2
39432025-03-25T22:34:34.227ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 244.963748ms iteration = 7 status = {"generation":2}
39442025-03-25T22:34:34.232ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:28:31.652727Z, time_modified: 2025-03-25T22:28:31.652727Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }) }
39452025-03-25T22:34:34.232ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 250.387137ms iteration = 7 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]}},"default_endpoint":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
39462025-03-25T22:34:34.250ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
39472025-03-25T22:34:34.250ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 267.207934ms iteration = 7 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
39482025-03-25T22:34:34.264ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 281.967132ms iteration = 14 status = {"physical_disks_added":0}
39492025-03-25T22:34:34.285ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
39502025-03-25T22:34:34.285ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
39512025-03-25T22:34:34.285ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ef97e45-9b96-4710-ba77-17ac667c93f8", "content-length": "26", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} })
39522025-03-25T22:34:34.285ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
39532025-03-25T22:34:34.287ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
39542025-03-25T22:34:34.306ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39552025-03-25T22:34:34.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39562025-03-25T22:34:34.308ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39572025-03-25T22:34:34.309ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
39582025-03-25T22:34:34.310ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "eeacb32e-16e9-481b-950e-8e01c2a5dbeb", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
39592025-03-25T22:34:34.310ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
39602025-03-25T22:34:34.310ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
39612025-03-25T22:34:34.310ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39622025-03-25T22:34:34.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39632025-03-25T22:34:34.311ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "492bed65-c297-452d-adbe-15befba867bf", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
39642025-03-25T22:34:34.311ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
39652025-03-25T22:34:34.312ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39662025-03-25T22:34:34.313ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
39672025-03-25T22:34:34.313ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 330.763253ms iteration = 9 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
39682025-03-25T22:34:34.314ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
39692025-03-25T22:34:34.314ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39702025-03-25T22:34:34.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39712025-03-25T22:34:34.352ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
39722025-03-25T22:34:34.380ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
39732025-03-25T22:34:34.380ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "67af22b7-1930-4639-aab6-20aaccf80b7f", "content-length": "1", "date": "Tue, 25 Mar 2025 22:34:33 GMT"} })
39742025-03-25T22:34:34.382ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
39752025-03-25T22:34:34.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
39762025-03-25T22:34:34.452ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 470.198256ms iteration = 13 status = {}
39772025-03-25T22:34:34.490ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
39782025-03-25T22:34:34.490ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "3267d299-ac60-4c94-bb7c-ec9ffbdea63f", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
39792025-03-25T22:34:34.490ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
39802025-03-25T22:34:34.492ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "2fd16663-d3a1-4e0f-9e34-307772e53b2d", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
39812025-03-25T22:34:34.492ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
39822025-03-25T22:34:34.493ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "18e116ce-742d-4472-a942-ad839a6f837f", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
39832025-03-25T22:34:34.493ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
39842025-03-25T22:34:34.495ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
39852025-03-25T22:34:34.495ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
39862025-03-25T22:34:34.495ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
39872025-03-25T22:34:34.495ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 512.931135ms iteration = 9 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
39882025-03-25T22:34:34.541ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
39892025-03-25T22:34:34.541ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
39902025-03-25T22:34:34.545ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 562.686273ms iteration = 13 status = null
39912025-03-25T22:34:34.587ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e842410-f28d-434b-baa5-b20c25651120", "content-length": "372", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} })
39922025-03-25T22:34:34.587ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39932025-03-25T22:34:34.588ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39942025-03-25T22:34:34.632ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
39952025-03-25T22:34:34.632ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
39962025-03-25T22:34:34.632ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "236c6de7-8e45-4e47-b3bb-fbac3eb8fa93", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} })
39972025-03-25T22:34:34.632ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39982025-03-25T22:34:34.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39992025-03-25T22:34:34.635ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
40002025-03-25T22:34:34.635ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40012025-03-25T22:34:34.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40022025-03-25T22:34:34.635ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "1edc44e6-5c07-4356-8948-f0dc1d544f6b", "content-length": "691", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} })
40032025-03-25T22:34:34.635ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
40042025-03-25T22:34:34.635ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
40052025-03-25T22:34:34.635ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
40062025-03-25T22:34:34.637ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 654.763732ms iteration = 13 status = {}
40072025-03-25T22:34:34.638ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40082025-03-25T22:34:34.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40092025-03-25T22:34:34.640ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
40102025-03-25T22:34:34.640ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
40112025-03-25T22:34:34.641ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "aefea5d5-f609-4dcd-86f8-a0a698edf90f", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} })
40122025-03-25T22:34:34.643ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 660.16218ms iteration = 13 status = {}
40132025-03-25T22:34:34.688ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
40142025-03-25T22:34:34.688ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:53650
40152025-03-25T22:34:34.688ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:53650 req_id = 4eae2ee7-c28f-43b7-969d-e2f16f991d68 roles = RoleSet { roles: {} } uri = /metrics/producers
40162025-03-25T22:34:34.689ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:53650 req_id = 4eae2ee7-c28f-43b7-969d-e2f16f991d68 resource = Database result = Ok(()) uri = /metrics/producers
40172025-03-25T22:34:34.724ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40182025-03-25T22:34:34.724ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40192025-03-25T22:34:34.726ZINFOnexus: assigned collector to new producer collector_id = 26728504-e36d-4baa-856d-cd0ce7a172a2 file = nexus/src/app/oximeter.rs:76 producer_id = 56ee8e9a-ccae-49c2-ac33-94d5159567dd
40202025-03-25T22:34:34.726ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 37766 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:53650 req_id = 4eae2ee7-c28f-43b7-969d-e2f16f991d68 response_code = 201 uri = /metrics/producers
40212025-03-25T22:34:34.726ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "4eae2ee7-c28f-43b7-969d-e2f16f991d68", "content-length": "41", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} })
40222025-03-25T22:34:34.726ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
40232025-03-25T22:34:34.726ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40242025-03-25T22:34:34.727ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40252025-03-25T22:34:34.728ZDEBGnexus (producer-registration-task): registered with nexus successfully
40262025-03-25T22:34:34.728ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
40272025-03-25T22:34:34.740ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
40282025-03-25T22:34:34.741ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "945d829f-421f-4b23-a21b-e4841297e08f", "content-length": "134", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} })
40292025-03-25T22:34:34.741ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
40302025-03-25T22:34:34.741ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
40312025-03-25T22:34:34.741ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
40322025-03-25T22:34:34.741ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
40332025-03-25T22:34:34.741ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
40342025-03-25T22:34:34.741ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
40352025-03-25T22:34:34.741ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
40362025-03-25T22:34:34.741ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
40372025-03-25T22:34:34.741ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
40382025-03-25T22:34:34.741ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
40392025-03-25T22:34:34.742ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
40402025-03-25T22:34:34.742ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40412025-03-25T22:34:34.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40422025-03-25T22:34:34.744ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
40432025-03-25T22:34:34.744ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "896aa891-8844-4f6e-afdd-55ee19723a2a", "content-length": "2", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} })
40442025-03-25T22:34:34.744ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
40452025-03-25T22:34:34.744ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1d2aa85-fe25-4cc5-99d0-5b9c46f1f19c", "content-length": "52", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} })
40462025-03-25T22:34:34.744ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
40472025-03-25T22:34:34.744ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
40482025-03-25T22:34:34.849ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "28f2a88e-1c2e-4d04-a1bb-9e839f6c55c8", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} })
40492025-03-25T22:34:34.849ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
40502025-03-25T22:34:34.849ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
40512025-03-25T22:34:34.850ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "895ca814-080e-49cc-8b48-9fc2de36749e", "content-length": "682", "date": "Tue, 25 Mar 2025 22:34:34 GMT"} })
40522025-03-25T22:34:34.850ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40532025-03-25T22:34:34.850ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40542025-03-25T22:34:34.854ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40552025-03-25T22:34:34.854ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40562025-03-25T22:34:34.858ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40572025-03-25T22:34:34.858ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40582025-03-25T22:34:34.859ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40592025-03-25T22:34:34.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40602025-03-25T22:34:34.866ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 883.089456ms iteration = 13 status = {}
40612025-03-25T22:34:36.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 257 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = eb4a7812-f49e-445f-b022-cedb9402c305 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
40622025-03-25T22:34:43.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
40632025-03-25T22:34:43.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 38 reason = Timeout
40642025-03-25T22:34:43.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40652025-03-25T22:34:43.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40662025-03-25T22:34:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40672025-03-25T22:34:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40682025-03-25T22:34:43.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40692025-03-25T22:34:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
40702025-03-25T22:34:43.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40712025-03-25T22:34:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40722025-03-25T22:34:43.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40732025-03-25T22:34:43.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40742025-03-25T22:34:43.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40752025-03-25T22:34:43.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
40762025-03-25T22:34:43.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40772025-03-25T22:34:43.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40782025-03-25T22:34:44.000ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
40792025-03-25T22:34:44.000ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.522119ms iteration = 38 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
40802025-03-25T22:34:44.014ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40812025-03-25T22:34:44.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40822025-03-25T22:34:44.052ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 69.316824ms iteration = 38 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
40832025-03-25T22:34:46.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 9f11bde7-dfa2-40c1-8a7c-73cf410ca10d roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
40842025-03-25T22:34:46.078ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 9f11bde7-dfa2-40c1-8a7c-73cf410ca10d resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
40852025-03-25T22:34:46.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3231 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 9f11bde7-dfa2-40c1-8a7c-73cf410ca10d response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
40862025-03-25T22:34:46.081ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 7ec9fa61-fb44-4b2b-a07e-1cc93efc8d54 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
40872025-03-25T22:34:46.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 7ec9fa61-fb44-4b2b-a07e-1cc93efc8d54 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
40882025-03-25T22:34:46.083ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2148 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 7ec9fa61-fb44-4b2b-a07e-1cc93efc8d54 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
40892025-03-25T22:34:46.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 276 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 4af8a8cc-b62a-4887-9750-787e6e188048 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
40902025-03-25T22:34:53.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
40912025-03-25T22:34:53.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 39 reason = Timeout
40922025-03-25T22:34:53.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40932025-03-25T22:34:53.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40942025-03-25T22:34:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40952025-03-25T22:34:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40962025-03-25T22:34:53.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40972025-03-25T22:34:53.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40982025-03-25T22:34:53.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40992025-03-25T22:34:53.986ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.518614ms iteration = 39 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
41002025-03-25T22:34:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
41012025-03-25T22:34:53.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41022025-03-25T22:34:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41032025-03-25T22:34:53.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41042025-03-25T22:34:53.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41052025-03-25T22:34:53.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41062025-03-25T22:34:53.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
41072025-03-25T22:34:53.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41082025-03-25T22:34:53.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41092025-03-25T22:34:54.002ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
41102025-03-25T22:34:54.002ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.361953ms iteration = 39 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
41112025-03-25T22:34:56.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 302 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = ce85ab1b-9cf4-41a3-93d4-1653437bbf72 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
41122025-03-25T22:35:03.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
41132025-03-25T22:35:03.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
41142025-03-25T22:35:03.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
41152025-03-25T22:35:03.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
41162025-03-25T22:35:03.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
41172025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
41182025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
41192025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 14 reason = Timeout
41202025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 40 reason = Timeout
41212025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 14 reason = Timeout
41222025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41232025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
41242025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
41252025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
41262025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
41272025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
41282025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 15 reason = Timeout
41292025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41302025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
41312025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
41322025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 14 reason = Timeout
41332025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41342025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
41352025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 14 reason = Timeout
41362025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 14 reason = Timeout
41372025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
41382025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 14 reason = Timeout
41392025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
41402025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
41412025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41422025-03-25T22:35:03.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 14 reason = Timeout
41432025-03-25T22:35:03.984ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
41442025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41452025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 14 reason = Timeout
41462025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
41472025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 14 reason = Timeout
41482025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
41492025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 14 reason = Timeout
41502025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
41512025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 14 reason = Timeout
41522025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
41532025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41542025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
41552025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
41562025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
41572025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
41582025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41592025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41602025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
41612025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
41622025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
41632025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41642025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
41652025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
41662025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
41672025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
41682025-03-25T22:35:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41692025-03-25T22:35:03.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1217, tv_nsec: 366581777 } })
41702025-03-25T22:35:03.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1217, tv_nsec: 366750580 } })
41712025-03-25T22:35:03.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
41722025-03-25T22:35:03.986ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41732025-03-25T22:35:03.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
41742025-03-25T22:35:03.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41752025-03-25T22:35:03.987ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.770377ms iteration = 14 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
41762025-03-25T22:35:03.987ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41772025-03-25T22:35:03.989ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41782025-03-25T22:35:03.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41792025-03-25T22:35:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
41802025-03-25T22:35:03.991ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
41812025-03-25T22:35:03.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
41822025-03-25T22:35:03.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
41832025-03-25T22:35:03.993ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41842025-03-25T22:35:03.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41852025-03-25T22:35:03.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
41862025-03-25T22:35:03.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41872025-03-25T22:35:03.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41882025-03-25T22:35:03.999ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41892025-03-25T22:35:03.999ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
41902025-03-25T22:35:04.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
41912025-03-25T22:35:04.000ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41922025-03-25T22:35:04.002ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
41932025-03-25T22:35:04.002ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41942025-03-25T22:35:04.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
41952025-03-25T22:35:04.002ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
41962025-03-25T22:35:04.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41972025-03-25T22:35:04.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
41982025-03-25T22:35:04.006ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41992025-03-25T22:35:04.007ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
42002025-03-25T22:35:04.008ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
42012025-03-25T22:35:04.008ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
42022025-03-25T22:35:04.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
42032025-03-25T22:35:04.009ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1217, tv_nsec: 389924283 } })
42042025-03-25T22:35:04.009ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
42052025-03-25T22:35:04.010ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42062025-03-25T22:35:04.010ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
42072025-03-25T22:35:04.010ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42082025-03-25T22:35:04.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42092025-03-25T22:35:04.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42102025-03-25T22:35:04.017ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
42112025-03-25T22:35:04.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
42122025-03-25T22:35:04.018ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42132025-03-25T22:35:04.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42142025-03-25T22:35:04.029ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42152025-03-25T22:35:04.031ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
42162025-03-25T22:35:04.031ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
42172025-03-25T22:35:04.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
42182025-03-25T22:35:04.045ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
42192025-03-25T22:35:04.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
42202025-03-25T22:35:04.051ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
42212025-03-25T22:35:04.051ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 67.575862ms iteration = 14 status = {"errors":[],"garbage_collect_requested":[]}
42222025-03-25T22:35:04.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
42232025-03-25T22:35:04.053ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 69.678351ms iteration = 14 status = {"errors":[],"finish_invoked_ok":[]}
42242025-03-25T22:35:04.054ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 70.962484ms iteration = 40 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
42252025-03-25T22:35:04.059ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
42262025-03-25T22:35:04.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
42272025-03-25T22:35:04.059ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
42282025-03-25T22:35:04.059ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
42292025-03-25T22:35:04.060ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6084041e-5de9-4ed7-8f23-74eeb1ff853e", "content-length": "26", "date": "Tue, 25 Mar 2025 22:35:04 GMT"} })
42302025-03-25T22:35:04.060ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
42312025-03-25T22:35:04.060ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 76.714369ms iteration = 14 status = {"errors":[],"requests_created_ok":[]}
42322025-03-25T22:35:04.061ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42332025-03-25T22:35:04.062ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
42342025-03-25T22:35:04.064ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
42352025-03-25T22:35:04.064ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
42362025-03-25T22:35:04.064ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42372025-03-25T22:35:04.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
42382025-03-25T22:35:04.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42392025-03-25T22:35:04.067ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 84.050013ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
42402025-03-25T22:35:04.070ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
42412025-03-25T22:35:04.070ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
42422025-03-25T22:35:04.070ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
42432025-03-25T22:35:04.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
42442025-03-25T22:35:04.073ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42452025-03-25T22:35:04.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42462025-03-25T22:35:04.077ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42472025-03-25T22:35:04.080ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
42482025-03-25T22:35:04.080ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42492025-03-25T22:35:04.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42502025-03-25T22:35:04.080ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42512025-03-25T22:35:04.081ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
42522025-03-25T22:35:04.081ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
42532025-03-25T22:35:04.081ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a472c55-de54-4864-bc14-c13543410e78", "content-length": "26", "date": "Tue, 25 Mar 2025 22:35:04 GMT"} })
42542025-03-25T22:35:04.081ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
42552025-03-25T22:35:04.083ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
42562025-03-25T22:35:04.084ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
42572025-03-25T22:35:04.084ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
42582025-03-25T22:35:04.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
42592025-03-25T22:35:04.087ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42602025-03-25T22:35:04.088ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
42612025-03-25T22:35:04.089ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
42622025-03-25T22:35:04.091ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
42632025-03-25T22:35:04.091ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42642025-03-25T22:35:04.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42652025-03-25T22:35:04.092ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
42662025-03-25T22:35:04.092ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 108.958357ms iteration = 40 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
42672025-03-25T22:35:04.114ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 130.016312ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
42682025-03-25T22:35:04.117ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
42692025-03-25T22:35:04.117ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
42702025-03-25T22:35:04.117ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 133.055496ms iteration = 14 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
42712025-03-25T22:35:04.117ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42722025-03-25T22:35:04.120ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
42732025-03-25T22:35:04.120ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42742025-03-25T22:35:04.121ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
42752025-03-25T22:35:04.121ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 137.227383ms iteration = 14 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
42762025-03-25T22:35:04.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42772025-03-25T22:35:04.121ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42782025-03-25T22:35:04.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42792025-03-25T22:35:04.123ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 139.595996ms iteration = 14 status = 0
42802025-03-25T22:35:04.130ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42812025-03-25T22:35:04.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42822025-03-25T22:35:04.131ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
42832025-03-25T22:35:04.131ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 147.51319ms iteration = 14 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
42842025-03-25T22:35:04.135ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 151.219248ms iteration = 14 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
42852025-03-25T22:35:04.167ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42862025-03-25T22:35:04.169ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
42872025-03-25T22:35:04.169ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42882025-03-25T22:35:04.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42892025-03-25T22:35:04.171ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42902025-03-25T22:35:04.171ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 187.918128ms iteration = 15 status = {"physical_disks_added":0}
42912025-03-25T22:35:04.173ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
42922025-03-25T22:35:04.173ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42932025-03-25T22:35:04.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42942025-03-25T22:35:04.178ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
42952025-03-25T22:35:04.178ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
42962025-03-25T22:35:04.179ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0846ce6c-0528-47e4-a8a7-0740e26055eb", "content-length": "26", "date": "Tue, 25 Mar 2025 22:35:04 GMT"} })
42972025-03-25T22:35:04.179ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
42982025-03-25T22:35:04.180ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
42992025-03-25T22:35:04.199ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 217.09876ms iteration = 14 status = {}
43002025-03-25T22:35:04.277ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
43012025-03-25T22:35:04.278ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4ccd152-0a2e-459b-993b-84bf02ddd6ec", "content-length": "1", "date": "Tue, 25 Mar 2025 22:35:03 GMT"} })
43022025-03-25T22:35:04.279ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
43032025-03-25T22:35:04.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
43042025-03-25T22:35:04.281ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 299.104425ms iteration = 14 status = null
43052025-03-25T22:35:04.319ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
43062025-03-25T22:35:04.319ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
43072025-03-25T22:35:04.319ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8ce9f58-5c32-470e-9666-d3642eeddb89", "content-length": "372", "date": "Tue, 25 Mar 2025 22:35:04 GMT"} })
43082025-03-25T22:35:04.319ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43092025-03-25T22:35:04.320ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43102025-03-25T22:35:04.331ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
43112025-03-25T22:35:04.332ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "55aa45b3-e94c-427c-b4b2-5f822c9dbfab", "content-length": "691", "date": "Tue, 25 Mar 2025 22:35:04 GMT"} })
43122025-03-25T22:35:04.332ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
43132025-03-25T22:35:04.332ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
43142025-03-25T22:35:04.332ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
43152025-03-25T22:35:04.334ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 350.075005ms iteration = 14 status = {}
43162025-03-25T22:35:04.378ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43172025-03-25T22:35:04.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43182025-03-25T22:35:04.381ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
43192025-03-25T22:35:04.381ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43202025-03-25T22:35:04.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43212025-03-25T22:35:04.390ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
43222025-03-25T22:35:04.390ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
43232025-03-25T22:35:04.390ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
43242025-03-25T22:35:04.390ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "b87fbd6f-aecc-4fd8-8ef4-5fcd0a72fe3a", "content-length": "134", "date": "Tue, 25 Mar 2025 22:35:03 GMT"} })
43252025-03-25T22:35:04.390ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
43262025-03-25T22:35:04.390ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
43272025-03-25T22:35:04.391ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
43282025-03-25T22:35:04.391ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
43292025-03-25T22:35:04.391ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
43302025-03-25T22:35:04.391ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
43312025-03-25T22:35:04.391ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
43322025-03-25T22:35:04.391ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
43332025-03-25T22:35:04.391ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
43342025-03-25T22:35:04.391ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
43352025-03-25T22:35:04.391ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
43362025-03-25T22:35:04.391ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43372025-03-25T22:35:04.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43382025-03-25T22:35:04.393ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
43392025-03-25T22:35:04.394ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dac144c-87f3-4a41-9540-9d3948c71402", "content-length": "2", "date": "Tue, 25 Mar 2025 22:35:03 GMT"} })
43402025-03-25T22:35:04.394ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
43412025-03-25T22:35:04.394ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad861337-dd85-4339-930e-9b02e21daea8", "content-length": "52", "date": "Tue, 25 Mar 2025 22:35:03 GMT"} })
43422025-03-25T22:35:04.394ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
43432025-03-25T22:35:04.394ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
43442025-03-25T22:35:04.470ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "5cbabf39-905f-4f21-8c42-da0a60e2bfb9", "date": "Tue, 25 Mar 2025 22:35:04 GMT"} })
43452025-03-25T22:35:04.470ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43462025-03-25T22:35:04.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43472025-03-25T22:35:04.473ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43482025-03-25T22:35:04.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43492025-03-25T22:35:04.477ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43502025-03-25T22:35:04.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43512025-03-25T22:35:04.481ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
43522025-03-25T22:35:04.481ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
43532025-03-25T22:35:04.481ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "f48862d4-3de2-4eb4-b587-fef07a1a5369", "date": "Tue, 25 Mar 2025 22:35:04 GMT"} })
43542025-03-25T22:35:04.488ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 504.601493ms iteration = 14 status = {}
43552025-03-25T22:35:04.584ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "ffbc9413-1d9a-4276-ac5c-b858fba31e74", "date": "Tue, 25 Mar 2025 22:35:04 GMT"} })
43562025-03-25T22:35:04.584ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
43572025-03-25T22:35:04.584ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
43582025-03-25T22:35:04.584ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "990ea036-4940-40b3-bac9-64fca74811f9", "content-length": "682", "date": "Tue, 25 Mar 2025 22:35:04 GMT"} })
43592025-03-25T22:35:04.584ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43602025-03-25T22:35:04.585ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43612025-03-25T22:35:04.588ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43622025-03-25T22:35:04.588ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43632025-03-25T22:35:04.592ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43642025-03-25T22:35:04.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43652025-03-25T22:35:04.594ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43662025-03-25T22:35:04.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43672025-03-25T22:35:04.602ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 618.198526ms iteration = 14 status = {}
43682025-03-25T22:35:06.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 278 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 0b699039-a24f-40a6-be7e-8d24fa1088bf response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
43692025-03-25T22:35:13.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
43702025-03-25T22:35:13.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 41 reason = Timeout
43712025-03-25T22:35:13.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43722025-03-25T22:35:13.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43732025-03-25T22:35:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43742025-03-25T22:35:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43752025-03-25T22:35:13.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43762025-03-25T22:35:13.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43772025-03-25T22:35:14.014ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43782025-03-25T22:35:14.017ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
43792025-03-25T22:35:14.017ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43802025-03-25T22:35:14.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43812025-03-25T22:35:14.019ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 35.958536ms iteration = 41 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
43822025-03-25T22:35:14.019ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43832025-03-25T22:35:14.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43842025-03-25T22:35:14.021ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43852025-03-25T22:35:14.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
43862025-03-25T22:35:14.024ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43872025-03-25T22:35:14.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43882025-03-25T22:35:14.033ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
43892025-03-25T22:35:14.033ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 50.057303ms iteration = 41 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
43902025-03-25T22:35:16.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 397991d0-65bd-4b0d-adb2-2ef3b6bff80b roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
43912025-03-25T22:35:16.078ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 397991d0-65bd-4b0d-adb2-2ef3b6bff80b resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
43922025-03-25T22:35:16.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3091 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 397991d0-65bd-4b0d-adb2-2ef3b6bff80b response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
43932025-03-25T22:35:16.081ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = f33fa7ed-36e0-404e-bcae-2f848379b7db roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
43942025-03-25T22:35:16.081ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = f33fa7ed-36e0-404e-bcae-2f848379b7db resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
43952025-03-25T22:35:16.083ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2268 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = f33fa7ed-36e0-404e-bcae-2f848379b7db response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
43962025-03-25T22:35:16.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 325 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 4375a14a-de9c-4998-8251-2b6d50a47cfa response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
43972025-03-25T22:35:23.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
43982025-03-25T22:35:23.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 42 reason = Timeout
43992025-03-25T22:35:23.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44002025-03-25T22:35:23.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44012025-03-25T22:35:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44022025-03-25T22:35:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44032025-03-25T22:35:23.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44042025-03-25T22:35:23.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44052025-03-25T22:35:23.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44062025-03-25T22:35:23.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.033084ms iteration = 42 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
44072025-03-25T22:35:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
44082025-03-25T22:35:23.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44092025-03-25T22:35:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44102025-03-25T22:35:23.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44112025-03-25T22:35:23.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44122025-03-25T22:35:23.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44132025-03-25T22:35:23.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
44142025-03-25T22:35:23.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44152025-03-25T22:35:23.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44162025-03-25T22:35:24.001ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
44172025-03-25T22:35:24.001ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.717821ms iteration = 42 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
44182025-03-25T22:35:26.085ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 271 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = c39e0c69-a354-480e-b83e-cd5f635d38d0 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
44192025-03-25T22:35:31.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 306c4f86-5b25-4703-a4bd-23bb876dc824 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
44202025-03-25T22:35:31.078ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 306c4f86-5b25-4703-a4bd-23bb876dc824 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
44212025-03-25T22:35:31.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3432 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 306c4f86-5b25-4703-a4bd-23bb876dc824 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
44222025-03-25T22:35:31.082ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = a178a6db-2933-44d2-a9be-6a3f342ee1f5 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
44232025-03-25T22:35:31.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = a178a6db-2933-44d2-a9be-6a3f342ee1f5 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
44242025-03-25T22:35:31.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2289 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = a178a6db-2933-44d2-a9be-6a3f342ee1f5 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
44252025-03-25T22:35:33.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
44262025-03-25T22:35:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
44272025-03-25T22:35:33.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
44282025-03-25T22:35:33.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
44292025-03-25T22:35:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
44302025-03-25T22:35:33.982ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
44312025-03-25T22:35:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
44322025-03-25T22:35:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
44332025-03-25T22:35:33.982ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 8 reason = Timeout
44342025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
44352025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
44362025-03-25T22:35:33.983ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:25:33.982933038 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
44372025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
44382025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
44392025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
44402025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
44412025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
44422025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
44432025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
44442025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
44452025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 8 reason = Timeout
44462025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
44472025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 15 reason = Timeout
44482025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 16 reason = Timeout
44492025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
44502025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
44512025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
44522025-03-25T22:35:33.983ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b
44532025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
44542025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 19.181\u{b5}s iteration = 9 status = {"enabled":false,"target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b"}
44552025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 15 reason = Timeout
44562025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
44572025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44582025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
44592025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 15 reason = Timeout
44602025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44612025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
44622025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 43 reason = Timeout
44632025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44642025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 15 reason = Timeout
44652025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 8 reason = Timeout
44662025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
44672025-03-25T22:35:33.983ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
44682025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 15 reason = Timeout
44692025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 15 reason = Timeout
44702025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
44712025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
44722025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44732025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 8 reason = Timeout
44742025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
44752025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 15 reason = Timeout
44762025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
44772025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
44782025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 8 reason = Timeout
44792025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
44802025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 15 reason = Timeout
44812025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
44822025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
44832025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
44842025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 15 reason = Timeout
44852025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
44862025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 15 reason = Timeout
44872025-03-25T22:35:33.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
44882025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
44892025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
44902025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 15 reason = Timeout
44912025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
44922025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
44932025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
44942025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
44952025-03-25T22:35:33.983ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8f1103be-ae21-497e-8fb0-5373f3346d6f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1247, tv_nsec: 364109232 } })
44962025-03-25T22:35:33.983ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("09fed906-08a5-4e51-9e85-fba7211b9594.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4ba21a7b-c053-4a77-807f-34395131fce2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1247, tv_nsec: 364120292 } })
44972025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
44982025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
44992025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
45002025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45012025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
45022025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
45032025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
45042025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
45052025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
45062025-03-25T22:35:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
45072025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
45082025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
45092025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
45102025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
45112025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
45122025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45132025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
45142025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
45152025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45162025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
45172025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
45182025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
45192025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
45202025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 980.458\u{b5}s iteration = 8 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
45212025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
45222025-03-25T22:35:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.263623ms iteration = 8 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
45232025-03-25T22:35:33.985ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
45242025-03-25T22:35:33.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
45252025-03-25T22:35:33.985ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45262025-03-25T22:35:33.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1247, tv_nsec: 367418092 } })
45272025-03-25T22:35:33.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1247, tv_nsec: 367593396 } })
45282025-03-25T22:35:33.987ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
45292025-03-25T22:35:33.987ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
45302025-03-25T22:35:33.987ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.398121ms iteration = 15 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
45312025-03-25T22:35:33.987ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
45322025-03-25T22:35:33.988ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:25:33.982933038 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
45332025-03-25T22:35:33.988ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
45342025-03-25T22:35:33.988ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.054422ms iteration = 8 status = {"expiration":"2025-03-25T22:25:33.982933038Z","pruned":[]}
45352025-03-25T22:35:33.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
45362025-03-25T22:35:33.988ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
45372025-03-25T22:35:33.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
45382025-03-25T22:35:33.989ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
45392025-03-25T22:35:33.989ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
45402025-03-25T22:35:33.989ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 6.632361ms iteration = 8 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
45412025-03-25T22:35:33.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45422025-03-25T22:35:33.991ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 8.727679ms iteration = 15 status = {"errors":[],"garbage_collect_requested":[]}
45432025-03-25T22:35:33.992ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45442025-03-25T22:35:33.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
45452025-03-25T22:35:33.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45462025-03-25T22:35:33.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45472025-03-25T22:35:33.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
45482025-03-25T22:35:33.994ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
45492025-03-25T22:35:33.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
45502025-03-25T22:35:33.995ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 12.024439ms iteration = 15 status = {"errors":[],"requests_created_ok":[]}
45512025-03-25T22:35:33.996ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
45522025-03-25T22:35:33.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
45532025-03-25T22:35:33.998ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45542025-03-25T22:35:34.000ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 16.830518ms iteration = 8 status = {"errors":[],"found_port_ok":[]}
45552025-03-25T22:35:34.000ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
45562025-03-25T22:35:34.000ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
45572025-03-25T22:35:34.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
45582025-03-25T22:35:34.006ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45592025-03-25T22:35:34.008ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45602025-03-25T22:35:34.010ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
45612025-03-25T22:35:34.010ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
45622025-03-25T22:35:34.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
45632025-03-25T22:35:34.011ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45642025-03-25T22:35:34.012ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
45652025-03-25T22:35:34.012ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45662025-03-25T22:35:34.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45672025-03-25T22:35:34.014ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
45682025-03-25T22:35:34.014ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
45692025-03-25T22:35:34.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
45702025-03-25T22:35:34.015ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
45712025-03-25T22:35:34.015ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
45722025-03-25T22:35:34.015ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
45732025-03-25T22:35:34.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
45742025-03-25T22:35:34.017ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45752025-03-25T22:35:34.018ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45762025-03-25T22:35:34.018ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 34.439798ms iteration = 15 status = {"errors":[],"finish_invoked_ok":[]}
45772025-03-25T22:35:34.018ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
45782025-03-25T22:35:34.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
45792025-03-25T22:35:34.018ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45802025-03-25T22:35:34.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45812025-03-25T22:35:34.022ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
45822025-03-25T22:35:34.023ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
45832025-03-25T22:35:34.032ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
45842025-03-25T22:35:34.032ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
45852025-03-25T22:35:34.032ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
45862025-03-25T22:35:34.032ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
45872025-03-25T22:35:34.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
45882025-03-25T22:35:34.033ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45892025-03-25T22:35:34.033ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1247, tv_nsec: 413665946 } })
45902025-03-25T22:35:34.033ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
45912025-03-25T22:35:34.034ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
45922025-03-25T22:35:34.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
45932025-03-25T22:35:34.036ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
45942025-03-25T22:35:34.036ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45952025-03-25T22:35:34.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45962025-03-25T22:35:34.038ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45972025-03-25T22:35:34.039ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
45982025-03-25T22:35:34.039ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 56.413738ms iteration = 15 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
45992025-03-25T22:35:34.041ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46002025-03-25T22:35:34.041ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46012025-03-25T22:35:34.044ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
46022025-03-25T22:35:34.044ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
46032025-03-25T22:35:34.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
46042025-03-25T22:35:34.045ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
46052025-03-25T22:35:34.045ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
46062025-03-25T22:35:34.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
46072025-03-25T22:35:34.046ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
46082025-03-25T22:35:34.046ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
46092025-03-25T22:35:34.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
46102025-03-25T22:35:34.057ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46112025-03-25T22:35:34.058ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
46122025-03-25T22:35:34.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
46132025-03-25T22:35:34.061ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
46142025-03-25T22:35:34.061ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
46152025-03-25T22:35:34.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46162025-03-25T22:35:34.064ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
46172025-03-25T22:35:34.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46182025-03-25T22:35:34.066ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46192025-03-25T22:35:34.069ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
46202025-03-25T22:35:34.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
46212025-03-25T22:35:34.070ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
46222025-03-25T22:35:34.070ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
46232025-03-25T22:35:34.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46242025-03-25T22:35:34.073ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
46252025-03-25T22:35:34.073ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
46262025-03-25T22:35:34.073ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
46272025-03-25T22:35:34.073ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
46282025-03-25T22:35:34.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
46292025-03-25T22:35:34.075ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46302025-03-25T22:35:34.077ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 93.858771ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
46312025-03-25T22:35:34.077ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46322025-03-25T22:35:34.080ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
46332025-03-25T22:35:34.081ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
46342025-03-25T22:35:34.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
46352025-03-25T22:35:34.082ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
46362025-03-25T22:35:34.082ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46372025-03-25T22:35:34.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46382025-03-25T22:35:34.086ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
46392025-03-25T22:35:34.086ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46402025-03-25T22:35:34.086ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
46412025-03-25T22:35:34.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46422025-03-25T22:35:34.087ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 104.351113ms iteration = 43 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
46432025-03-25T22:35:34.093ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
46442025-03-25T22:35:34.093ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
46452025-03-25T22:35:34.093ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 110.170329ms iteration = 8 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
46462025-03-25T22:35:34.107ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
46472025-03-25T22:35:34.107ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 123.812058ms iteration = 15 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
46482025-03-25T22:35:34.107ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
46492025-03-25T22:35:34.107ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
46502025-03-25T22:35:34.107ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
46512025-03-25T22:35:34.108ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
46522025-03-25T22:35:34.109ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46532025-03-25T22:35:34.112ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
46542025-03-25T22:35:34.113ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
46552025-03-25T22:35:34.113ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46562025-03-25T22:35:34.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46572025-03-25T22:35:34.122ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
46582025-03-25T22:35:34.122ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
46592025-03-25T22:35:34.123ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
46602025-03-25T22:35:34.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
46612025-03-25T22:35:34.125ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
46622025-03-25T22:35:34.125ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 0 version = 2
46632025-03-25T22:35:34.125ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
46642025-03-25T22:35:34.125ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
46652025-03-25T22:35:34.126ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
46662025-03-25T22:35:34.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
46672025-03-25T22:35:34.127ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46682025-03-25T22:35:34.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46692025-03-25T22:35:34.130ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46702025-03-25T22:35:34.131ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46712025-03-25T22:35:34.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46722025-03-25T22:35:34.135ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
46732025-03-25T22:35:34.135ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
46742025-03-25T22:35:34.135ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
46752025-03-25T22:35:34.135ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 152.255126ms iteration = 43 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
46762025-03-25T22:35:34.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
46772025-03-25T22:35:34.135ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
46782025-03-25T22:35:34.135ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
46792025-03-25T22:35:34.135ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
46802025-03-25T22:35:34.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
46812025-03-25T22:35:34.137ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46822025-03-25T22:35:34.137ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 1 version = 2
46832025-03-25T22:35:34.137ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
46842025-03-25T22:35:34.137ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:29:07.710706 UTC dns_group = external generation = 2
46852025-03-25T22:35:34.137ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 154.5998ms iteration = 8 status = {"generation":2}
46862025-03-25T22:35:34.137ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 154.255143ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
46872025-03-25T22:35:34.140ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
46882025-03-25T22:35:34.140ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
46892025-03-25T22:35:34.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
46902025-03-25T22:35:34.140ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
46912025-03-25T22:35:34.140ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "13a95ea9-ccb9-4fdf-806b-e4315e11c928", "content-length": "26", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} })
46922025-03-25T22:35:34.140ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
46932025-03-25T22:35:34.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
46942025-03-25T22:35:34.143ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
46952025-03-25T22:35:34.143ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
46962025-03-25T22:35:34.143ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
46972025-03-25T22:35:34.143ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:29:06.674111 UTC dns_group = internal generation = 1
46982025-03-25T22:35:34.143ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 160.605329ms iteration = 8 status = {"generation":1}
46992025-03-25T22:35:34.158ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
47002025-03-25T22:35:34.158ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
47012025-03-25T22:35:34.158ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "48dca91a-04d1-423a-92b4-382c4da34d00", "content-length": "26", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} })
47022025-03-25T22:35:34.158ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
47032025-03-25T22:35:34.160ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
47042025-03-25T22:35:34.160ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
47052025-03-25T22:35:34.163ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
47062025-03-25T22:35:34.163ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
47072025-03-25T22:35:34.163ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 179.901761ms iteration = 15 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
47082025-03-25T22:35:34.173ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47092025-03-25T22:35:34.175ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
47102025-03-25T22:35:34.175ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
47112025-03-25T22:35:34.175ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
47122025-03-25T22:35:34.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
47132025-03-25T22:35:34.175ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47142025-03-25T22:35:34.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
47152025-03-25T22:35:34.178ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
47162025-03-25T22:35:34.178ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
47172025-03-25T22:35:34.178ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
47182025-03-25T22:35:34.181ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 198.45626ms iteration = 15 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
47192025-03-25T22:35:34.185ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
47202025-03-25T22:35:34.185ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
47212025-03-25T22:35:34.185ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fd26367-38a3-4d48-a632-d84ca2323a4f", "content-length": "26", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} })
47222025-03-25T22:35:34.185ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
47232025-03-25T22:35:34.188ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
47242025-03-25T22:35:34.213ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 230.485234ms iteration = 16 status = {"physical_disks_added":0}
47252025-03-25T22:35:34.238ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
47262025-03-25T22:35:34.238ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 254.96126ms iteration = 8 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
47272025-03-25T22:35:34.387ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
47282025-03-25T22:35:34.387ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "2e587512-29fb-4a66-b177-2067315e0aee", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
47292025-03-25T22:35:34.387ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
47302025-03-25T22:35:34.389ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "5c1816ee-0e37-40bb-bed8-26beb7c2b458", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
47312025-03-25T22:35:34.389ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
47322025-03-25T22:35:34.391ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
47332025-03-25T22:35:34.391ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 408.846306ms iteration = 10 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
47342025-03-25T22:35:34.410ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
47352025-03-25T22:35:34.411ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b1d34e7-eb3b-47a8-9fce-695b596d1971", "content-length": "1", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} })
47362025-03-25T22:35:34.412ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
47372025-03-25T22:35:34.412ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
47382025-03-25T22:35:34.412ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
47392025-03-25T22:35:34.413ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "9de3678a-f0c8-4991-b52b-6923d30d42da", "content-length": "372", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} })
47402025-03-25T22:35:34.413ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
47412025-03-25T22:35:34.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
47422025-03-25T22:35:34.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
47432025-03-25T22:35:34.416ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 433.540377ms iteration = 15 status = 0
47442025-03-25T22:35:34.417ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 434.571636ms iteration = 15 status = null
47452025-03-25T22:35:34.454ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 471.945597ms iteration = 15 status = {}
47462025-03-25T22:35:34.466ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:28:31.652727Z, time_modified: 2025-03-25T22:28:31.652727Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }) }
47472025-03-25T22:35:34.466ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 483.516188ms iteration = 8 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]}},"default_endpoint":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
47482025-03-25T22:35:34.469ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
47492025-03-25T22:35:34.488ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
47502025-03-25T22:35:34.488ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
47512025-03-25T22:35:34.489ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "c258dc4c-8bde-4e92-b5f0-e87d653e0e2d", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} })
47522025-03-25T22:35:34.489ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
47532025-03-25T22:35:34.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
47542025-03-25T22:35:34.494ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
47552025-03-25T22:35:34.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
47562025-03-25T22:35:34.497ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
47572025-03-25T22:35:34.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
47582025-03-25T22:35:34.501ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
47592025-03-25T22:35:34.501ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
47602025-03-25T22:35:34.501ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "a067542e-8032-4a8c-ba70-40a3a40f8acd", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} })
47612025-03-25T22:35:34.503ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 520.227179ms iteration = 15 status = {}
47622025-03-25T22:35:34.511ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
47632025-03-25T22:35:34.513ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "db9376a4-b162-41f6-9ba1-7c2d7fe84cfa", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
47642025-03-25T22:35:34.513ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
47652025-03-25T22:35:34.514ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "fd259c65-8744-4961-aa24-b25ba5d21753", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
47662025-03-25T22:35:34.514ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
47672025-03-25T22:35:34.516ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "a33c1211-38d5-4ca9-8fe7-ccc795fbe433", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
47682025-03-25T22:35:34.516ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
47692025-03-25T22:35:34.518ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
47702025-03-25T22:35:34.518ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 535.553438ms iteration = 10 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
47712025-03-25T22:35:34.528ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47722025-03-25T22:35:34.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47732025-03-25T22:35:34.530ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
47742025-03-25T22:35:34.530ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47752025-03-25T22:35:34.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47762025-03-25T22:35:34.536ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
47772025-03-25T22:35:34.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
47782025-03-25T22:35:34.537ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47792025-03-25T22:35:34.539ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
47802025-03-25T22:35:34.539ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
47812025-03-25T22:35:34.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
47822025-03-25T22:35:34.541ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47832025-03-25T22:35:34.543ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
47842025-03-25T22:35:34.543ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
47852025-03-25T22:35:34.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
47862025-03-25T22:35:34.543ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
47872025-03-25T22:35:34.543ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "e070b7ec-b044-43f6-8a39-fbb16e362451", "content-length": "134", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} })
47882025-03-25T22:35:34.543ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
47892025-03-25T22:35:34.543ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
47902025-03-25T22:35:34.544ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
47912025-03-25T22:35:34.544ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
47922025-03-25T22:35:34.544ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47932025-03-25T22:35:34.544ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47942025-03-25T22:35:34.544ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47952025-03-25T22:35:34.544ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
47962025-03-25T22:35:34.544ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47972025-03-25T22:35:34.544ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
47982025-03-25T22:35:34.544ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
47992025-03-25T22:35:34.544ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48002025-03-25T22:35:34.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48012025-03-25T22:35:34.546ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
48022025-03-25T22:35:34.547ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "34c73311-2bef-4b50-a646-b03dab12684b", "content-length": "2", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} })
48032025-03-25T22:35:34.547ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
48042025-03-25T22:35:34.547ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "872b0d12-9021-4e8a-a7e3-1540c9e8418d", "content-length": "52", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} })
48052025-03-25T22:35:34.547ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
48062025-03-25T22:35:34.547ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
48072025-03-25T22:35:34.578ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
48082025-03-25T22:35:34.621ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e59b0c7a-577c-4148-9134-4a8535a780b1", "content-length": "691", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} })
48092025-03-25T22:35:34.621ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
48102025-03-25T22:35:34.621ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
48112025-03-25T22:35:34.621ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
48122025-03-25T22:35:34.622ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 639.653186ms iteration = 15 status = {}
48132025-03-25T22:35:34.726ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "d250c862-ed7d-4154-aa36-872641f78260", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} })
48142025-03-25T22:35:34.726ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
48152025-03-25T22:35:34.726ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
48162025-03-25T22:35:34.726ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a07ce84-5424-4d30-9763-2b94c91fb135", "content-length": "682", "date": "Tue, 25 Mar 2025 22:35:34 GMT"} })
48172025-03-25T22:35:34.726ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48182025-03-25T22:35:34.727ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48192025-03-25T22:35:34.730ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48202025-03-25T22:35:34.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48212025-03-25T22:35:34.733ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48222025-03-25T22:35:34.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48232025-03-25T22:35:34.736ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48242025-03-25T22:35:34.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48252025-03-25T22:35:34.743ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 760.233065ms iteration = 15 status = {}
48262025-03-25T22:35:36.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 292 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 614783a8-4323-4fc8-b975-85e0927ffc1b response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
48272025-03-25T22:35:43.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 44 reason = Timeout
48282025-03-25T22:35:43.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
48292025-03-25T22:35:43.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48302025-03-25T22:35:43.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48312025-03-25T22:35:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48322025-03-25T22:35:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48332025-03-25T22:35:43.984ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48342025-03-25T22:35:43.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48352025-03-25T22:35:44.013ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48362025-03-25T22:35:44.014ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.547946ms iteration = 44 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
48372025-03-25T22:35:44.016ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
48382025-03-25T22:35:44.016ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48392025-03-25T22:35:44.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48402025-03-25T22:35:44.018ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48412025-03-25T22:35:44.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48422025-03-25T22:35:44.019ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48432025-03-25T22:35:44.022ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
48442025-03-25T22:35:44.022ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48452025-03-25T22:35:44.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48462025-03-25T22:35:44.031ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
48472025-03-25T22:35:44.031ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 48.530365ms iteration = 44 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
48482025-03-25T22:35:46.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 8cff3133-70a6-4d60-9bea-22ecbadefffe roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
48492025-03-25T22:35:46.078ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 8cff3133-70a6-4d60-9bea-22ecbadefffe resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
48502025-03-25T22:35:46.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3560 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 8cff3133-70a6-4d60-9bea-22ecbadefffe response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
48512025-03-25T22:35:46.081ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 5c51f7d0-dfa9-408a-a24a-fa95920c57c6 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
48522025-03-25T22:35:46.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 5c51f7d0-dfa9-408a-a24a-fa95920c57c6 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
48532025-03-25T22:35:46.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 404 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 6583bcbe-b2fe-4de1-b00c-916a5d03f722 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
48542025-03-25T22:35:46.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2501 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 5c51f7d0-dfa9-408a-a24a-fa95920c57c6 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
48552025-03-25T22:35:53.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 45 reason = Timeout
48562025-03-25T22:35:53.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48572025-03-25T22:35:53.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
48582025-03-25T22:35:53.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48592025-03-25T22:35:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48602025-03-25T22:35:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48612025-03-25T22:35:53.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48622025-03-25T22:35:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
48632025-03-25T22:35:53.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48642025-03-25T22:35:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48652025-03-25T22:35:53.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48662025-03-25T22:35:53.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48672025-03-25T22:35:54.017ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48682025-03-25T22:35:54.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48692025-03-25T22:35:54.019ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 35.754882ms iteration = 45 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
48702025-03-25T22:35:54.026ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48712025-03-25T22:35:54.029ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
48722025-03-25T22:35:54.029ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48732025-03-25T22:35:54.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48742025-03-25T22:35:54.039ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
48752025-03-25T22:35:54.039ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 55.716136ms iteration = 45 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
48762025-03-25T22:35:56.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 422 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 05386c99-a735-4237-ba3d-f60f7546d0e0 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
48772025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
48782025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
48792025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
48802025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
48812025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
48822025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
48832025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
48842025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 16 reason = Timeout
48852025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
48862025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 16 reason = Timeout
48872025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
48882025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 16 reason = Timeout
48892025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 16 reason = Timeout
48902025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
48912025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
48922025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
48932025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 17 reason = Timeout
48942025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
48952025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
48962025-03-25T22:36:03.982ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 16 reason = Timeout
48972025-03-25T22:36:03.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
48982025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48992025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 16 reason = Timeout
49002025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
49012025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
49022025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 46 reason = Timeout
49032025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49042025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 16 reason = Timeout
49052025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 16 reason = Timeout
49062025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
49072025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
49082025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
49092025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
49102025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49112025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 16 reason = Timeout
49122025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49132025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
49142025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 16 reason = Timeout
49152025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
49162025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 16 reason = Timeout
49172025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49182025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
49192025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
49202025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
49212025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
49222025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49232025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
49242025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49252025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
49262025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
49272025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
49282025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
49292025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49302025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
49312025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49322025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
49332025-03-25T22:36:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49342025-03-25T22:36:03.985ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.173548ms iteration = 16 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
49352025-03-25T22:36:03.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1277, tv_nsec: 366385163 } })
49362025-03-25T22:36:03.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1277, tv_nsec: 366522106 } })
49372025-03-25T22:36:03.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
49382025-03-25T22:36:03.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
49392025-03-25T22:36:03.988ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
49402025-03-25T22:36:03.988ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49412025-03-25T22:36:03.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
49422025-03-25T22:36:03.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49432025-03-25T22:36:03.991ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49442025-03-25T22:36:03.993ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49452025-03-25T22:36:03.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49462025-03-25T22:36:03.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
49472025-03-25T22:36:03.994ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
49482025-03-25T22:36:03.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
49492025-03-25T22:36:03.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
49502025-03-25T22:36:03.997ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49512025-03-25T22:36:03.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49522025-03-25T22:36:03.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
49532025-03-25T22:36:03.999ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49542025-03-25T22:36:03.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49552025-03-25T22:36:04.001ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49562025-03-25T22:36:04.004ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
49572025-03-25T22:36:04.004ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49582025-03-25T22:36:04.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49592025-03-25T22:36:04.016ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1277, tv_nsec: 397201006 } })
49602025-03-25T22:36:04.017ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
49612025-03-25T22:36:04.017ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49622025-03-25T22:36:04.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49632025-03-25T22:36:04.022ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 39.532481ms iteration = 46 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
49642025-03-25T22:36:04.029ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49652025-03-25T22:36:04.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49662025-03-25T22:36:04.031ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49672025-03-25T22:36:04.036ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
49682025-03-25T22:36:04.036ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49692025-03-25T22:36:04.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49702025-03-25T22:36:04.044ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 61.174805ms iteration = 16 status = {"errors":[],"requests_created_ok":[]}
49712025-03-25T22:36:04.045ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
49722025-03-25T22:36:04.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
49732025-03-25T22:36:04.047ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 64.359974ms iteration = 16 status = {"errors":[],"finish_invoked_ok":[]}
49742025-03-25T22:36:04.048ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
49752025-03-25T22:36:04.048ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 65.81992ms iteration = 46 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
49762025-03-25T22:36:04.055ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 72.721926ms iteration = 16 status = {"errors":[],"garbage_collect_requested":[]}
49772025-03-25T22:36:04.055ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
49782025-03-25T22:36:04.055ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
49792025-03-25T22:36:04.055ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
49802025-03-25T22:36:04.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
49812025-03-25T22:36:04.058ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49822025-03-25T22:36:04.062ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
49832025-03-25T22:36:04.062ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
49842025-03-25T22:36:04.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
49852025-03-25T22:36:04.066ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
49862025-03-25T22:36:04.066ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
49872025-03-25T22:36:04.066ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
49882025-03-25T22:36:04.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
49892025-03-25T22:36:04.066ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
49902025-03-25T22:36:04.066ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49912025-03-25T22:36:04.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
49922025-03-25T22:36:04.067ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
49932025-03-25T22:36:04.067ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
49942025-03-25T22:36:04.067ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 85.073642ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
49952025-03-25T22:36:04.069ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
49962025-03-25T22:36:04.069ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49972025-03-25T22:36:04.069ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
49982025-03-25T22:36:04.069ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49992025-03-25T22:36:04.069ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50002025-03-25T22:36:04.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50012025-03-25T22:36:04.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
50022025-03-25T22:36:04.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50032025-03-25T22:36:04.072ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
50042025-03-25T22:36:04.072ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
50052025-03-25T22:36:04.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
50062025-03-25T22:36:04.074ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
50072025-03-25T22:36:04.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
50082025-03-25T22:36:04.075ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50092025-03-25T22:36:04.075ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50102025-03-25T22:36:04.076ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50112025-03-25T22:36:04.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50122025-03-25T22:36:04.077ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
50132025-03-25T22:36:04.077ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50142025-03-25T22:36:04.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50152025-03-25T22:36:04.079ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
50162025-03-25T22:36:04.079ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
50172025-03-25T22:36:04.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
50182025-03-25T22:36:04.082ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
50192025-03-25T22:36:04.082ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 99.320881ms iteration = 16 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
50202025-03-25T22:36:04.101ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 118.256957ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
50212025-03-25T22:36:04.105ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50222025-03-25T22:36:04.107ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
50232025-03-25T22:36:04.107ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50242025-03-25T22:36:04.108ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50252025-03-25T22:36:04.115ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 132.436045ms iteration = 16 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
50262025-03-25T22:36:04.116ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
50272025-03-25T22:36:04.116ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
50282025-03-25T22:36:04.116ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 133.79744ms iteration = 16 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
50292025-03-25T22:36:04.123ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50302025-03-25T22:36:04.126ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
50312025-03-25T22:36:04.126ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50322025-03-25T22:36:04.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50332025-03-25T22:36:04.131ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
50342025-03-25T22:36:04.131ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 148.137862ms iteration = 16 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
50352025-03-25T22:36:04.144ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
50362025-03-25T22:36:04.144ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
50372025-03-25T22:36:04.144ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dc938f5-bd26-4df7-9adb-40f7c151a94b", "content-length": "26", "date": "Tue, 25 Mar 2025 22:36:04 GMT"} })
50382025-03-25T22:36:04.144ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
50392025-03-25T22:36:04.147ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
50402025-03-25T22:36:04.150ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
50412025-03-25T22:36:04.150ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
50422025-03-25T22:36:04.151ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a95718ee-0db0-40ed-b5d6-408d8451c6f4", "content-length": "26", "date": "Tue, 25 Mar 2025 22:36:04 GMT"} })
50432025-03-25T22:36:04.151ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
50442025-03-25T22:36:04.152ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
50452025-03-25T22:36:04.158ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
50462025-03-25T22:36:04.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
50472025-03-25T22:36:04.187ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
50482025-03-25T22:36:04.187ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
50492025-03-25T22:36:04.187ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "67c4fa86-201f-4f57-af7f-447111c229fa", "content-length": "372", "date": "Tue, 25 Mar 2025 22:36:04 GMT"} })
50502025-03-25T22:36:04.187ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50512025-03-25T22:36:04.188ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50522025-03-25T22:36:04.188ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
50532025-03-25T22:36:04.188ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
50542025-03-25T22:36:04.188ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "39981c78-dbe4-4020-af6d-64a408cbe0d9", "content-length": "26", "date": "Tue, 25 Mar 2025 22:36:04 GMT"} })
50552025-03-25T22:36:04.189ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
50562025-03-25T22:36:04.189ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 206.66231ms iteration = 16 status = 0
50572025-03-25T22:36:04.190ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
50582025-03-25T22:36:04.236ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 254.282888ms iteration = 16 status = {}
50592025-03-25T22:36:04.244ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
50602025-03-25T22:36:04.244ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
50612025-03-25T22:36:04.245ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "b205bd94-b45c-4bb1-a69f-071e5adcf8b6", "date": "Tue, 25 Mar 2025 22:36:04 GMT"} })
50622025-03-25T22:36:04.245ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50632025-03-25T22:36:04.245ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50642025-03-25T22:36:04.247ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50652025-03-25T22:36:04.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50662025-03-25T22:36:04.250ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50672025-03-25T22:36:04.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50682025-03-25T22:36:04.254ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
50692025-03-25T22:36:04.254ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
50702025-03-25T22:36:04.254ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "8297f63c-9c37-42ea-b629-9db7f3d8b61a", "date": "Tue, 25 Mar 2025 22:36:04 GMT"} })
50712025-03-25T22:36:04.256ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 273.258074ms iteration = 16 status = {}
50722025-03-25T22:36:04.259ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
50732025-03-25T22:36:04.260ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "0891cf6e-e14d-4c64-84b1-8d355b39cdde", "content-length": "1", "date": "Tue, 25 Mar 2025 22:36:03 GMT"} })
50742025-03-25T22:36:04.261ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
50752025-03-25T22:36:04.261ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
50762025-03-25T22:36:04.263ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 281.461344ms iteration = 16 status = null
50772025-03-25T22:36:04.300ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 317.352708ms iteration = 17 status = {"physical_disks_added":0}
50782025-03-25T22:36:04.305ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
50792025-03-25T22:36:04.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
50802025-03-25T22:36:04.307ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50812025-03-25T22:36:04.309ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
50822025-03-25T22:36:04.309ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
50832025-03-25T22:36:04.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
50842025-03-25T22:36:04.313ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50852025-03-25T22:36:04.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50862025-03-25T22:36:04.316ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
50872025-03-25T22:36:04.316ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50882025-03-25T22:36:04.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50892025-03-25T22:36:04.336ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
50902025-03-25T22:36:04.337ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "3aa4af8c-aab8-4bef-a071-ec15e9a393b0", "content-length": "134", "date": "Tue, 25 Mar 2025 22:36:04 GMT"} })
50912025-03-25T22:36:04.337ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
50922025-03-25T22:36:04.337ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
50932025-03-25T22:36:04.337ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
50942025-03-25T22:36:04.337ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
50952025-03-25T22:36:04.337ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
50962025-03-25T22:36:04.337ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
50972025-03-25T22:36:04.337ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
50982025-03-25T22:36:04.337ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
50992025-03-25T22:36:04.337ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
51002025-03-25T22:36:04.337ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
51012025-03-25T22:36:04.337ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
51022025-03-25T22:36:04.338ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51032025-03-25T22:36:04.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51042025-03-25T22:36:04.341ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
51052025-03-25T22:36:04.342ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5e8b77c-8d69-4de9-bac6-7494467ab9a8", "content-length": "2", "date": "Tue, 25 Mar 2025 22:36:04 GMT"} })
51062025-03-25T22:36:04.342ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
51072025-03-25T22:36:04.342ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b32d49d-9689-49bf-afa0-44991cd4bf9d", "content-length": "52", "date": "Tue, 25 Mar 2025 22:36:04 GMT"} })
51082025-03-25T22:36:04.342ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
51092025-03-25T22:36:04.342ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
51102025-03-25T22:36:04.350ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51112025-03-25T22:36:04.352ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
51122025-03-25T22:36:04.352ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
51132025-03-25T22:36:04.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
51142025-03-25T22:36:04.396ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
51152025-03-25T22:36:04.463ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a9c4acc-b91b-4469-bfff-85b9baa475c6", "content-length": "691", "date": "Tue, 25 Mar 2025 22:36:04 GMT"} })
51162025-03-25T22:36:04.464ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
51172025-03-25T22:36:04.464ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
51182025-03-25T22:36:04.464ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
51192025-03-25T22:36:04.465ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 482.54979ms iteration = 16 status = {}
51202025-03-25T22:36:04.571ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "2501a866-4e31-4d5c-bbcb-cd19a39065f1", "date": "Tue, 25 Mar 2025 22:36:04 GMT"} })
51212025-03-25T22:36:04.571ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
51222025-03-25T22:36:04.571ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
51232025-03-25T22:36:04.571ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c66d5f0d-89f4-408c-a273-e2e7b3054fba", "content-length": "682", "date": "Tue, 25 Mar 2025 22:36:04 GMT"} })
51242025-03-25T22:36:04.571ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51252025-03-25T22:36:04.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51262025-03-25T22:36:04.575ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51272025-03-25T22:36:04.575ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51282025-03-25T22:36:04.578ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51292025-03-25T22:36:04.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51302025-03-25T22:36:04.580ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51312025-03-25T22:36:04.580ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51322025-03-25T22:36:04.587ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 604.18ms iteration = 16 status = {}
51332025-03-25T22:36:06.085ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 264 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = de377c70-7eb8-4e39-969e-0545be85270b response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
51342025-03-25T22:36:13.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
51352025-03-25T22:36:13.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 47 reason = Timeout
51362025-03-25T22:36:13.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51372025-03-25T22:36:13.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51382025-03-25T22:36:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51392025-03-25T22:36:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51402025-03-25T22:36:13.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51412025-03-25T22:36:13.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
51422025-03-25T22:36:13.987ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51432025-03-25T22:36:13.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51442025-03-25T22:36:13.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51452025-03-25T22:36:13.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51462025-03-25T22:36:13.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51472025-03-25T22:36:13.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
51482025-03-25T22:36:13.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51492025-03-25T22:36:13.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51502025-03-25T22:36:14.002ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
51512025-03-25T22:36:14.002ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.359423ms iteration = 47 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
51522025-03-25T22:36:14.014ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51532025-03-25T22:36:14.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51542025-03-25T22:36:14.045ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.045281ms iteration = 47 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
51552025-03-25T22:36:16.077ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = dd63708c-ae78-4bac-9451-cc3e4a09de28 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
51562025-03-25T22:36:16.078ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = dd63708c-ae78-4bac-9451-cc3e4a09de28 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
51572025-03-25T22:36:16.080ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2853 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = dd63708c-ae78-4bac-9451-cc3e4a09de28 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
51582025-03-25T22:36:16.081ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 9fb79505-e684-48da-b371-153bccd614a3 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
51592025-03-25T22:36:16.081ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 9fb79505-e684-48da-b371-153bccd614a3 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
51602025-03-25T22:36:16.082ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 1804 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 9fb79505-e684-48da-b371-153bccd614a3 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
51612025-03-25T22:36:16.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 229 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 0056e28a-bdd0-4224-8862-fb5751320981 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
51622025-03-25T22:36:23.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 48 reason = Timeout
51632025-03-25T22:36:23.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
51642025-03-25T22:36:23.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51652025-03-25T22:36:23.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51662025-03-25T22:36:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51672025-03-25T22:36:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51682025-03-25T22:36:24.012ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51692025-03-25T22:36:24.014ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51702025-03-25T22:36:24.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51712025-03-25T22:36:24.015ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
51722025-03-25T22:36:24.015ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51732025-03-25T22:36:24.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51742025-03-25T22:36:24.016ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.47934ms iteration = 48 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
51752025-03-25T22:36:24.017ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51762025-03-25T22:36:24.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51772025-03-25T22:36:24.019ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51782025-03-25T22:36:24.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
51792025-03-25T22:36:24.021ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51802025-03-25T22:36:24.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51812025-03-25T22:36:24.031ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
51822025-03-25T22:36:24.031ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 48.003396ms iteration = 48 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
51832025-03-25T22:36:26.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 302 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 969b5e17-cb4b-43c8-9069-9e93eda0a61a response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
51842025-03-25T22:36:31.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 1f2e568e-5dc9-42ee-b700-44de2947eff2 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
51852025-03-25T22:36:31.078ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 1f2e568e-5dc9-42ee-b700-44de2947eff2 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
51862025-03-25T22:36:31.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3289 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 1f2e568e-5dc9-42ee-b700-44de2947eff2 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
51872025-03-25T22:36:31.082ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 3aea5b5e-985d-46bf-936d-1ac432eb90a7 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
51882025-03-25T22:36:31.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 3aea5b5e-985d-46bf-936d-1ac432eb90a7 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
51892025-03-25T22:36:31.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2571 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 3aea5b5e-985d-46bf-936d-1ac432eb90a7 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
51902025-03-25T22:36:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
51912025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
51922025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
51932025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
51942025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 9 reason = Timeout
51952025-03-25T22:36:33.983ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:26:33.982961398 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
51962025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
51972025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
51982025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
51992025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
52002025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
52012025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
52022025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
52032025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
52042025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
52052025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
52062025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
52072025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
52082025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
52092025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
52102025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 17 reason = Timeout
52112025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
52122025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 17 reason = Timeout
52132025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 17 reason = Timeout
52142025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
52152025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
52162025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 17 reason = Timeout
52172025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52182025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 17 reason = Timeout
52192025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 49 reason = Timeout
52202025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
52212025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 17 reason = Timeout
52222025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
52232025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52242025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 17 reason = Timeout
52252025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
52262025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
52272025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 17 reason = Timeout
52282025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
52292025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 9 reason = Timeout
52302025-03-25T22:36:33.983ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
52312025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
52322025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 9 reason = Timeout
52332025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
52342025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
52352025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 9 reason = Timeout
52362025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52372025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
52382025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
52392025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 17 reason = Timeout
52402025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
52412025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
52422025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 17 reason = Timeout
52432025-03-25T22:36:33.983ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b
52442025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
52452025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
52462025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.421\u{b5}s iteration = 10 status = {"enabled":false,"target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b"}
52472025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 18 reason = Timeout
52482025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
52492025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
52502025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
52512025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 17 reason = Timeout
52522025-03-25T22:36:33.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
52532025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52542025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52552025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 9 reason = Timeout
52562025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
52572025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
52582025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
52592025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
52602025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
52612025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
52622025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
52632025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52642025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
52652025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52662025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
52672025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
52682025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
52692025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
52702025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
52712025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
52722025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52732025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52742025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
52752025-03-25T22:36:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
52762025-03-25T22:36:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
52772025-03-25T22:36:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
52782025-03-25T22:36:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
52792025-03-25T22:36:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
52802025-03-25T22:36:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
52812025-03-25T22:36:33.984ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("09fed906-08a5-4e51-9e85-fba7211b9594.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4ba21a7b-c053-4a77-807f-34395131fce2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1307, tv_nsec: 364613721 } })
52822025-03-25T22:36:33.984ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8f1103be-ae21-497e-8fb0-5373f3346d6f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1307, tv_nsec: 364618471 } })
52832025-03-25T22:36:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52842025-03-25T22:36:33.984ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
52852025-03-25T22:36:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.503187ms iteration = 9 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
52862025-03-25T22:36:33.984ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
52872025-03-25T22:36:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.745432ms iteration = 9 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
52882025-03-25T22:36:33.986ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.991905ms iteration = 17 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
52892025-03-25T22:36:33.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1307, tv_nsec: 366938804 } })
52902025-03-25T22:36:33.986ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
52912025-03-25T22:36:33.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1307, tv_nsec: 366997274 } })
52922025-03-25T22:36:33.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
52932025-03-25T22:36:33.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
52942025-03-25T22:36:33.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
52952025-03-25T22:36:33.987ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
52962025-03-25T22:36:33.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
52972025-03-25T22:36:33.988ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:26:33.982961398 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
52982025-03-25T22:36:33.988ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.614322ms iteration = 9 status = {"expiration":"2025-03-25T22:26:33.982961398Z","pruned":[]}
52992025-03-25T22:36:33.988ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53002025-03-25T22:36:33.991ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
53012025-03-25T22:36:33.991ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
53022025-03-25T22:36:33.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
53032025-03-25T22:36:34.010ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 27.090714ms iteration = 9 status = {"errors":[],"found_port_ok":[]}
53042025-03-25T22:36:34.019ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53052025-03-25T22:36:34.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53062025-03-25T22:36:34.057ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53072025-03-25T22:36:34.057ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 74.704933ms iteration = 17 status = {"errors":[],"requests_created_ok":[]}
53082025-03-25T22:36:34.060ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 77.204608ms iteration = 17 status = {"errors":[],"garbage_collect_requested":[]}
53092025-03-25T22:36:34.061ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
53102025-03-25T22:36:34.061ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
53112025-03-25T22:36:34.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
53122025-03-25T22:36:34.062ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53132025-03-25T22:36:34.063ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 79.836956ms iteration = 17 status = {"errors":[],"finish_invoked_ok":[]}
53142025-03-25T22:36:34.063ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53152025-03-25T22:36:34.065ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
53162025-03-25T22:36:34.065ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
53172025-03-25T22:36:34.066ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
53182025-03-25T22:36:34.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
53192025-03-25T22:36:34.066ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
53202025-03-25T22:36:34.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
53212025-03-25T22:36:34.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
53222025-03-25T22:36:34.068ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
53232025-03-25T22:36:34.068ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
53242025-03-25T22:36:34.068ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
53252025-03-25T22:36:34.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
53262025-03-25T22:36:34.069ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
53272025-03-25T22:36:34.074ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
53282025-03-25T22:36:34.074ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53292025-03-25T22:36:34.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
53302025-03-25T22:36:34.076ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53312025-03-25T22:36:34.076ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 93.644868ms iteration = 49 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
53322025-03-25T22:36:34.079ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
53332025-03-25T22:36:34.079ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
53342025-03-25T22:36:34.080ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
53352025-03-25T22:36:34.080ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
53362025-03-25T22:36:34.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
53372025-03-25T22:36:34.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
53382025-03-25T22:36:34.081ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1307, tv_nsec: 462103629 } })
53392025-03-25T22:36:34.082ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
53402025-03-25T22:36:34.083ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
53412025-03-25T22:36:34.083ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
53422025-03-25T22:36:34.083ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 100.134786ms iteration = 9 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
53432025-03-25T22:36:34.086ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53442025-03-25T22:36:34.086ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53452025-03-25T22:36:34.086ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
53462025-03-25T22:36:34.086ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
53472025-03-25T22:36:34.086ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
53482025-03-25T22:36:34.086ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
53492025-03-25T22:36:34.087ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53502025-03-25T22:36:34.088ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
53512025-03-25T22:36:34.088ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
53522025-03-25T22:36:34.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
53532025-03-25T22:36:34.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
53542025-03-25T22:36:34.089ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
53552025-03-25T22:36:34.089ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53562025-03-25T22:36:34.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53572025-03-25T22:36:34.090ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
53582025-03-25T22:36:34.090ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
53592025-03-25T22:36:34.090ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
53602025-03-25T22:36:34.090ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
53612025-03-25T22:36:34.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53622025-03-25T22:36:34.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
53632025-03-25T22:36:34.093ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 110.490855ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
53642025-03-25T22:36:34.093ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
53652025-03-25T22:36:34.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
53662025-03-25T22:36:34.094ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53672025-03-25T22:36:34.095ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53682025-03-25T22:36:34.098ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
53692025-03-25T22:36:34.098ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53702025-03-25T22:36:34.098ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
53712025-03-25T22:36:34.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53722025-03-25T22:36:34.098ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
53732025-03-25T22:36:34.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
53742025-03-25T22:36:34.100ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
53752025-03-25T22:36:34.100ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53762025-03-25T22:36:34.100ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53772025-03-25T22:36:34.101ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
53782025-03-25T22:36:34.101ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
53792025-03-25T22:36:34.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
53802025-03-25T22:36:34.106ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
53812025-03-25T22:36:34.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
53822025-03-25T22:36:34.109ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53832025-03-25T22:36:34.112ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
53842025-03-25T22:36:34.112ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
53852025-03-25T22:36:34.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
53862025-03-25T22:36:34.130ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
53872025-03-25T22:36:34.130ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
53882025-03-25T22:36:34.131ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c445f0a8-4a2f-4b0c-b952-5dcd10906fe0", "content-length": "26", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} })
53892025-03-25T22:36:34.131ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
53902025-03-25T22:36:34.133ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53912025-03-25T22:36:34.137ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
53922025-03-25T22:36:34.138ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
53932025-03-25T22:36:34.138ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
53942025-03-25T22:36:34.138ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 155.361483ms iteration = 17 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
53952025-03-25T22:36:34.140ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
53962025-03-25T22:36:34.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53972025-03-25T22:36:34.143ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
53982025-03-25T22:36:34.143ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
53992025-03-25T22:36:34.143ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
54002025-03-25T22:36:34.143ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
54012025-03-25T22:36:34.144ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
54022025-03-25T22:36:34.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
54032025-03-25T22:36:34.145ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
54042025-03-25T22:36:34.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
54052025-03-25T22:36:34.146ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
54062025-03-25T22:36:34.146ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54072025-03-25T22:36:34.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
54082025-03-25T22:36:34.148ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54092025-03-25T22:36:34.149ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
54102025-03-25T22:36:34.149ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
54112025-03-25T22:36:34.151ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
54122025-03-25T22:36:34.151ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
54132025-03-25T22:36:34.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
54142025-03-25T22:36:34.152ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 168.799198ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
54152025-03-25T22:36:34.153ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
54162025-03-25T22:36:34.153ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
54172025-03-25T22:36:34.153ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 170.412898ms iteration = 9 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
54182025-03-25T22:36:34.153ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
54192025-03-25T22:36:34.153ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
54202025-03-25T22:36:34.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
54212025-03-25T22:36:34.158ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54222025-03-25T22:36:34.158ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
54232025-03-25T22:36:34.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
54242025-03-25T22:36:34.160ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
54252025-03-25T22:36:34.160ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 177.724481ms iteration = 17 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
54262025-03-25T22:36:34.162ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
54272025-03-25T22:36:34.162ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54282025-03-25T22:36:34.162ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54292025-03-25T22:36:34.162ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
54302025-03-25T22:36:34.162ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
54312025-03-25T22:36:34.163ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 180.508993ms iteration = 17 status = 0
54322025-03-25T22:36:34.163ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54332025-03-25T22:36:34.166ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
54342025-03-25T22:36:34.166ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
54352025-03-25T22:36:34.166ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
54362025-03-25T22:36:34.168ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54372025-03-25T22:36:34.171ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
54382025-03-25T22:36:34.171ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
54392025-03-25T22:36:34.172ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
54402025-03-25T22:36:34.177ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
54412025-03-25T22:36:34.177ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 193.53633ms iteration = 49 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
54422025-03-25T22:36:34.191ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
54432025-03-25T22:36:34.191ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
54442025-03-25T22:36:34.191ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5618839d-4bb3-4ec9-baff-1465d66386e9", "content-length": "26", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} })
54452025-03-25T22:36:34.191ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
54462025-03-25T22:36:34.193ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
54472025-03-25T22:36:34.198ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
54482025-03-25T22:36:34.198ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
54492025-03-25T22:36:34.198ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
54502025-03-25T22:36:34.198ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
54512025-03-25T22:36:34.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
54522025-03-25T22:36:34.200ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54532025-03-25T22:36:34.203ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
54542025-03-25T22:36:34.203ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
54552025-03-25T22:36:34.203ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
54562025-03-25T22:36:34.204ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
54572025-03-25T22:36:34.206ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
54582025-03-25T22:36:34.206ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 222.952766ms iteration = 17 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
54592025-03-25T22:36:34.208ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
54602025-03-25T22:36:34.208ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 0 version = 2
54612025-03-25T22:36:34.208ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
54622025-03-25T22:36:34.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
54632025-03-25T22:36:34.209ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54642025-03-25T22:36:34.211ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
54652025-03-25T22:36:34.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
54662025-03-25T22:36:34.212ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
54672025-03-25T22:36:34.212ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
54682025-03-25T22:36:34.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
54692025-03-25T22:36:34.213ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54702025-03-25T22:36:34.213ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 230.333011ms iteration = 17 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
54712025-03-25T22:36:34.214ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 1 version = 2
54722025-03-25T22:36:34.214ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
54732025-03-25T22:36:34.214ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:29:07.710706 UTC dns_group = external generation = 2
54742025-03-25T22:36:34.214ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 231.682945ms iteration = 9 status = {"generation":2}
54752025-03-25T22:36:34.215ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
54762025-03-25T22:36:34.215ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
54772025-03-25T22:36:34.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
54782025-03-25T22:36:34.231ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
54792025-03-25T22:36:34.231ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
54802025-03-25T22:36:34.231ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "da15509b-ed4d-4434-916f-a9bb14980fa1", "content-length": "26", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} })
54812025-03-25T22:36:34.231ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
54822025-03-25T22:36:34.232ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:28:31.652727Z, time_modified: 2025-03-25T22:28:31.652727Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }) }
54832025-03-25T22:36:34.232ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 249.533231ms iteration = 9 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]}},"default_endpoint":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
54842025-03-25T22:36:34.233ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
54852025-03-25T22:36:34.244ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
54862025-03-25T22:36:34.244ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
54872025-03-25T22:36:34.244ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
54882025-03-25T22:36:34.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
54892025-03-25T22:36:34.246ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54902025-03-25T22:36:34.250ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
54912025-03-25T22:36:34.250ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
54922025-03-25T22:36:34.250ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
54932025-03-25T22:36:34.253ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
54942025-03-25T22:36:34.253ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
54952025-03-25T22:36:34.253ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:29:06.674111 UTC dns_group = internal generation = 1
54962025-03-25T22:36:34.253ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 270.469373ms iteration = 9 status = {"generation":1}
54972025-03-25T22:36:34.253ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 270.286899ms iteration = 18 status = {"physical_disks_added":0}
54982025-03-25T22:36:34.269ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54992025-03-25T22:36:34.271ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
55002025-03-25T22:36:34.271ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
55012025-03-25T22:36:34.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
55022025-03-25T22:36:34.278ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
55032025-03-25T22:36:34.278ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 294.759336ms iteration = 9 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
55042025-03-25T22:36:34.286ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
55052025-03-25T22:36:34.406ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
55062025-03-25T22:36:34.407ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b58fcd5-c58f-49f5-a96c-f8831cf0661e", "content-length": "1", "date": "Tue, 25 Mar 2025 22:36:33 GMT"} })
55072025-03-25T22:36:34.409ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
55082025-03-25T22:36:34.409ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
55092025-03-25T22:36:34.411ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 428.917703ms iteration = 17 status = null
55102025-03-25T22:36:34.428ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
55112025-03-25T22:36:34.436ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 453.40845ms iteration = 17 status = {}
55122025-03-25T22:36:34.489ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
55132025-03-25T22:36:34.489ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
55142025-03-25T22:36:34.560ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
55152025-03-25T22:36:34.560ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "c7104f19-df5d-42bb-9575-145bf2f1af1c", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
55162025-03-25T22:36:34.561ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
55172025-03-25T22:36:34.562ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "fb688fa9-0809-4f5b-9ee3-9ddc8402a2fd", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
55182025-03-25T22:36:34.562ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
55192025-03-25T22:36:34.563ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
55202025-03-25T22:36:34.563ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 580.591109ms iteration = 11 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
55212025-03-25T22:36:34.570ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
55222025-03-25T22:36:34.570ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "b0d60885-3246-40ba-921c-bdba4a0c7d8a", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
55232025-03-25T22:36:34.571ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
55242025-03-25T22:36:34.572ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "22f4b345-41e6-41c4-a1ac-62c5efc9882f", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
55252025-03-25T22:36:34.572ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
55262025-03-25T22:36:34.573ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "6acef7c7-f841-44b9-9052-99b9198e2f80", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
55272025-03-25T22:36:34.573ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
55282025-03-25T22:36:34.575ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
55292025-03-25T22:36:34.575ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 592.620538ms iteration = 11 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
55302025-03-25T22:36:34.580ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "31d1286b-c7bf-44ef-851d-c72a1ce301fb", "content-length": "372", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} })
55312025-03-25T22:36:34.580ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
55322025-03-25T22:36:34.580ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
55332025-03-25T22:36:34.580ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
55342025-03-25T22:36:34.581ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3d325d2-0893-4f61-8106-823edd2caffb", "content-length": "691", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} })
55352025-03-25T22:36:34.581ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
55362025-03-25T22:36:34.581ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
55372025-03-25T22:36:34.581ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
55382025-03-25T22:36:34.582ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 599.82079ms iteration = 17 status = {}
55392025-03-25T22:36:34.606ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55402025-03-25T22:36:34.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55412025-03-25T22:36:34.609ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
55422025-03-25T22:36:34.609ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55432025-03-25T22:36:34.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55442025-03-25T22:36:34.622ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
55452025-03-25T22:36:34.623ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
55462025-03-25T22:36:34.623ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "c4610975-4882-4798-9deb-b218c910aed4", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} })
55472025-03-25T22:36:34.623ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
55482025-03-25T22:36:34.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
55492025-03-25T22:36:34.625ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
55502025-03-25T22:36:34.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
55512025-03-25T22:36:34.626ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
55522025-03-25T22:36:34.627ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "436a2c1e-bd93-4518-8963-f9fe628e37a8", "content-length": "134", "date": "Tue, 25 Mar 2025 22:36:33 GMT"} })
55532025-03-25T22:36:34.627ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
55542025-03-25T22:36:34.627ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
55552025-03-25T22:36:34.627ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
55562025-03-25T22:36:34.627ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
55572025-03-25T22:36:34.627ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
55582025-03-25T22:36:34.627ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
55592025-03-25T22:36:34.627ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
55602025-03-25T22:36:34.627ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
55612025-03-25T22:36:34.627ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
55622025-03-25T22:36:34.627ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
55632025-03-25T22:36:34.627ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
55642025-03-25T22:36:34.628ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55652025-03-25T22:36:34.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55662025-03-25T22:36:34.628ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
55672025-03-25T22:36:34.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
55682025-03-25T22:36:34.629ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
55692025-03-25T22:36:34.629ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bf87b99-69f3-4965-8500-46f375bd3ea4", "content-length": "2", "date": "Tue, 25 Mar 2025 22:36:33 GMT"} })
55702025-03-25T22:36:34.629ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
55712025-03-25T22:36:34.630ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "eef00b0d-e58f-4960-9dd1-2ab19c302b72", "content-length": "52", "date": "Tue, 25 Mar 2025 22:36:33 GMT"} })
55722025-03-25T22:36:34.630ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
55732025-03-25T22:36:34.630ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
55742025-03-25T22:36:34.632ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
55752025-03-25T22:36:34.632ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
55762025-03-25T22:36:34.732ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "1f69e4a2-95b5-4e51-b017-7d7e060b863f", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} })
55772025-03-25T22:36:34.732ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
55782025-03-25T22:36:34.732ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
55792025-03-25T22:36:34.732ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "64cf7dca-ee23-4d57-a9cb-50860ba3f7b7", "content-length": "682", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} })
55802025-03-25T22:36:34.732ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55812025-03-25T22:36:34.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55822025-03-25T22:36:34.733ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "4972cd0a-bdd9-4189-8bc0-a9220ccae030", "date": "Tue, 25 Mar 2025 22:36:34 GMT"} })
55832025-03-25T22:36:34.734ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 751.208641ms iteration = 17 status = {}
55842025-03-25T22:36:34.736ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55852025-03-25T22:36:34.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55862025-03-25T22:36:34.739ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55872025-03-25T22:36:34.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55882025-03-25T22:36:34.740ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55892025-03-25T22:36:34.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55902025-03-25T22:36:34.748ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 765.034803ms iteration = 17 status = {}
55912025-03-25T22:36:36.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 315 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 896a8dd8-7bf6-48c1-b988-724cb1d6bb87 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
55922025-03-25T22:36:43.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 50 reason = Timeout
55932025-03-25T22:36:43.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
55942025-03-25T22:36:43.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55952025-03-25T22:36:43.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55962025-03-25T22:36:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55972025-03-25T22:36:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55982025-03-25T22:36:43.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55992025-03-25T22:36:43.989ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56002025-03-25T22:36:43.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56012025-03-25T22:36:43.989ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
56022025-03-25T22:36:43.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56032025-03-25T22:36:43.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56042025-03-25T22:36:43.991ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 8.048647ms iteration = 50 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
56052025-03-25T22:36:43.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56062025-03-25T22:36:43.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56072025-03-25T22:36:43.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56082025-03-25T22:36:43.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
56092025-03-25T22:36:43.997ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56102025-03-25T22:36:43.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56112025-03-25T22:36:44.006ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
56122025-03-25T22:36:44.006ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.908408ms iteration = 50 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
56132025-03-25T22:36:46.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 261 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 755499e8-e659-4f75-ac9e-e7a387ca2a11 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
56142025-03-25T22:36:53.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 51 reason = Timeout
56152025-03-25T22:36:53.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
56162025-03-25T22:36:53.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56172025-03-25T22:36:53.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56182025-03-25T22:36:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56192025-03-25T22:36:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56202025-03-25T22:36:53.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56212025-03-25T22:36:53.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56222025-03-25T22:36:53.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56232025-03-25T22:36:53.986ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.729588ms iteration = 51 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
56242025-03-25T22:36:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
56252025-03-25T22:36:53.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56262025-03-25T22:36:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56272025-03-25T22:36:53.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56282025-03-25T22:36:53.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56292025-03-25T22:36:53.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56302025-03-25T22:36:53.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
56312025-03-25T22:36:53.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56322025-03-25T22:36:53.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56332025-03-25T22:36:54.003ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
56342025-03-25T22:36:54.003ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.137137ms iteration = 51 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
56352025-03-25T22:36:56.085ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 429 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 9ad1ce9e-3416-4adf-9068-eeb14eb5c189 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
56362025-03-25T22:37:01.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = ef5fd110-9e51-4196-94ab-b176ba9680ee roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
56372025-03-25T22:37:01.079ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = ef5fd110-9e51-4196-94ab-b176ba9680ee resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
56382025-03-25T22:37:01.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2728 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = ef5fd110-9e51-4196-94ab-b176ba9680ee response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
56392025-03-25T22:37:01.081ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 8c45ebbe-ecad-4dfd-a607-032b3a374eb0 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
56402025-03-25T22:37:01.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 8c45ebbe-ecad-4dfd-a607-032b3a374eb0 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
56412025-03-25T22:37:01.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2566 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56582 req_id = 8c45ebbe-ecad-4dfd-a607-032b3a374eb0 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
56422025-03-25T22:37:03.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
56432025-03-25T22:37:03.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
56442025-03-25T22:37:03.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
56452025-03-25T22:37:03.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
56462025-03-25T22:37:03.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
56472025-03-25T22:37:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
56482025-03-25T22:37:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
56492025-03-25T22:37:03.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 18 reason = Timeout
56502025-03-25T22:37:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 52 reason = Timeout
56512025-03-25T22:37:03.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 18 reason = Timeout
56522025-03-25T22:37:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56532025-03-25T22:37:03.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56542025-03-25T22:37:03.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
56552025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
56562025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 19 reason = Timeout
56572025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
56582025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56592025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56602025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56612025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 18 reason = Timeout
56622025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
56632025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 18 reason = Timeout
56642025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 18 reason = Timeout
56652025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56662025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
56672025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 18 reason = Timeout
56682025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
56692025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
56702025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
56712025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
56722025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 18 reason = Timeout
56732025-03-25T22:37:03.984ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
56742025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56752025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 18 reason = Timeout
56762025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
56772025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 18 reason = Timeout
56782025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
56792025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 18 reason = Timeout
56802025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
56812025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 18 reason = Timeout
56822025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
56832025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56842025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56852025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56862025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
56872025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56882025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
56892025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
56902025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
56912025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56922025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56932025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
56942025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56952025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
56962025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
56972025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
56982025-03-25T22:37:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
56992025-03-25T22:37:03.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1337, tv_nsec: 366739830 } })
57002025-03-25T22:37:03.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
57012025-03-25T22:37:03.986ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
57022025-03-25T22:37:03.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
57032025-03-25T22:37:03.987ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.677305ms iteration = 18 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
57042025-03-25T22:37:03.987ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1337, tv_nsec: 368069194 } })
57052025-03-25T22:37:03.987ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57062025-03-25T22:37:03.987ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57072025-03-25T22:37:03.987ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
57082025-03-25T22:37:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
57092025-03-25T22:37:03.991ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
57102025-03-25T22:37:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
57112025-03-25T22:37:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
57122025-03-25T22:37:03.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57132025-03-25T22:37:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57142025-03-25T22:37:03.995ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57152025-03-25T22:37:03.996ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57162025-03-25T22:37:03.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
57172025-03-25T22:37:03.998ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
57182025-03-25T22:37:03.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
57192025-03-25T22:37:03.999ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57202025-03-25T22:37:04.000ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
57212025-03-25T22:37:04.000ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57222025-03-25T22:37:04.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57232025-03-25T22:37:04.001ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57242025-03-25T22:37:04.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
57252025-03-25T22:37:04.003ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
57262025-03-25T22:37:04.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
57272025-03-25T22:37:04.004ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
57282025-03-25T22:37:04.004ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57292025-03-25T22:37:04.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57302025-03-25T22:37:04.008ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
57312025-03-25T22:37:04.008ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
57322025-03-25T22:37:04.008ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
57332025-03-25T22:37:04.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
57342025-03-25T22:37:04.019ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57352025-03-25T22:37:04.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57362025-03-25T22:37:04.019ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57372025-03-25T22:37:04.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57382025-03-25T22:37:04.021ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57392025-03-25T22:37:04.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57402025-03-25T22:37:04.021ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
57412025-03-25T22:37:04.022ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1337, tv_nsec: 402597464 } })
57422025-03-25T22:37:04.022ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57432025-03-25T22:37:04.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
57442025-03-25T22:37:04.022ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
57452025-03-25T22:37:04.025ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
57462025-03-25T22:37:04.025ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 41.658029ms iteration = 18 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
57472025-03-25T22:37:04.026ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
57482025-03-25T22:37:04.026ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57492025-03-25T22:37:04.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57502025-03-25T22:37:04.031ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57512025-03-25T22:37:04.034ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
57522025-03-25T22:37:04.034ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
57532025-03-25T22:37:04.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
57542025-03-25T22:37:04.044ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57552025-03-25T22:37:04.046ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
57562025-03-25T22:37:04.046ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 62.817376ms iteration = 52 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
57572025-03-25T22:37:04.047ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
57582025-03-25T22:37:04.047ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57592025-03-25T22:37:04.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57602025-03-25T22:37:04.048ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
57612025-03-25T22:37:04.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
57622025-03-25T22:37:04.051ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 67.254456ms iteration = 18 status = {"errors":[],"requests_created_ok":[]}
57632025-03-25T22:37:04.055ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
57642025-03-25T22:37:04.055ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57652025-03-25T22:37:04.055ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4353072-d1b4-4e34-b15e-fab0dfa367d3", "content-length": "26", "date": "Tue, 25 Mar 2025 22:37:04 GMT"} })
57662025-03-25T22:37:04.055ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57672025-03-25T22:37:04.057ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
57682025-03-25T22:37:04.057ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
57692025-03-25T22:37:04.057ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57702025-03-25T22:37:04.058ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "46d5aae5-5ed8-4f6d-80c5-d7abeb7a5008", "content-length": "26", "date": "Tue, 25 Mar 2025 22:37:04 GMT"} })
57712025-03-25T22:37:04.058ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57722025-03-25T22:37:04.060ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
57732025-03-25T22:37:04.065ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 81.395714ms iteration = 18 status = {"errors":[],"finish_invoked_ok":[]}
57742025-03-25T22:37:04.067ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57752025-03-25T22:37:04.067ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57762025-03-25T22:37:04.075ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57772025-03-25T22:37:04.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57782025-03-25T22:37:04.077ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 93.659738ms iteration = 18 status = {"errors":[],"garbage_collect_requested":[]}
57792025-03-25T22:37:04.078ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57802025-03-25T22:37:04.078ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
57812025-03-25T22:37:04.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
57822025-03-25T22:37:04.080ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
57832025-03-25T22:37:04.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
57842025-03-25T22:37:04.082ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
57852025-03-25T22:37:04.082ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57862025-03-25T22:37:04.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57872025-03-25T22:37:04.084ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
57882025-03-25T22:37:04.084ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57892025-03-25T22:37:04.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
57902025-03-25T22:37:04.087ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
57912025-03-25T22:37:04.087ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
57922025-03-25T22:37:04.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
57932025-03-25T22:37:04.088ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 104.618598ms iteration = 18 status = 0
57942025-03-25T22:37:04.088ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 104.681489ms iteration = 52 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
57952025-03-25T22:37:04.094ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
57962025-03-25T22:37:04.094ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
57972025-03-25T22:37:04.094ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
57982025-03-25T22:37:04.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
57992025-03-25T22:37:04.100ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
58002025-03-25T22:37:04.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
58012025-03-25T22:37:04.106ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58022025-03-25T22:37:04.108ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
58032025-03-25T22:37:04.108ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58042025-03-25T22:37:04.108ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58052025-03-25T22:37:04.131ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 147.795886ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
58062025-03-25T22:37:04.136ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 152.009572ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
58072025-03-25T22:37:04.137ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
58082025-03-25T22:37:04.137ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
58092025-03-25T22:37:04.137ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a0065f7-12cb-46e1-b815-7188ed8d7498", "content-length": "26", "date": "Tue, 25 Mar 2025 22:37:04 GMT"} })
58102025-03-25T22:37:04.137ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
58112025-03-25T22:37:04.139ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
58122025-03-25T22:37:04.141ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
58132025-03-25T22:37:04.141ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 157.630845ms iteration = 18 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
58142025-03-25T22:37:04.144ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 160.820383ms iteration = 18 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
58152025-03-25T22:37:04.148ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
58162025-03-25T22:37:04.148ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
58172025-03-25T22:37:04.148ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 164.080453ms iteration = 18 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
58182025-03-25T22:37:04.155ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 173.108597ms iteration = 18 status = {}
58192025-03-25T22:37:04.167ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
58202025-03-25T22:37:04.168ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca72efb9-80fb-423f-97fa-5d17e6da7be2", "content-length": "1", "date": "Tue, 25 Mar 2025 22:37:03 GMT"} })
58212025-03-25T22:37:04.169ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
58222025-03-25T22:37:04.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
58232025-03-25T22:37:04.171ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 188.779774ms iteration = 18 status = null
58242025-03-25T22:37:04.177ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 193.268105ms iteration = 19 status = {"physical_disks_added":0}
58252025-03-25T22:37:04.190ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58262025-03-25T22:37:04.191ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58272025-03-25T22:37:04.191ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
58282025-03-25T22:37:04.191ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
58292025-03-25T22:37:04.192ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c46ba75-b4cb-411c-9a9f-88866e04e3e9", "content-length": "372", "date": "Tue, 25 Mar 2025 22:37:04 GMT"} })
58302025-03-25T22:37:04.192ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58312025-03-25T22:37:04.192ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58322025-03-25T22:37:04.192ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58332025-03-25T22:37:04.194ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
58342025-03-25T22:37:04.195ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58352025-03-25T22:37:04.195ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58362025-03-25T22:37:04.196ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58372025-03-25T22:37:04.198ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
58382025-03-25T22:37:04.198ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58392025-03-25T22:37:04.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58402025-03-25T22:37:04.246ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
58412025-03-25T22:37:04.246ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
58422025-03-25T22:37:04.246ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "bfaa2ac4-117b-4493-a2bc-a85313287eb1", "date": "Tue, 25 Mar 2025 22:37:04 GMT"} })
58432025-03-25T22:37:04.246ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58442025-03-25T22:37:04.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58452025-03-25T22:37:04.249ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58462025-03-25T22:37:04.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58472025-03-25T22:37:04.253ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58482025-03-25T22:37:04.253ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58492025-03-25T22:37:04.256ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
58502025-03-25T22:37:04.256ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
58512025-03-25T22:37:04.256ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "2492c3c6-6325-4e9f-bc81-60e24ed72bc8", "date": "Tue, 25 Mar 2025 22:37:04 GMT"} })
58522025-03-25T22:37:04.258ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 273.931126ms iteration = 18 status = {}
58532025-03-25T22:37:04.273ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58542025-03-25T22:37:04.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58552025-03-25T22:37:04.275ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
58562025-03-25T22:37:04.275ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58572025-03-25T22:37:04.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58582025-03-25T22:37:04.300ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
58592025-03-25T22:37:04.311ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
58602025-03-25T22:37:04.311ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "649c2699-4f55-4fba-94d3-e0a1a79bd224", "content-length": "134", "date": "Tue, 25 Mar 2025 22:37:03 GMT"} })
58612025-03-25T22:37:04.311ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
58622025-03-25T22:37:04.311ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
58632025-03-25T22:37:04.311ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
58642025-03-25T22:37:04.311ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
58652025-03-25T22:37:04.311ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58662025-03-25T22:37:04.311ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58672025-03-25T22:37:04.311ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58682025-03-25T22:37:04.311ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
58692025-03-25T22:37:04.311ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58702025-03-25T22:37:04.311ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
58712025-03-25T22:37:04.312ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
58722025-03-25T22:37:04.312ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58732025-03-25T22:37:04.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58742025-03-25T22:37:04.315ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
58752025-03-25T22:37:04.315ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "877eec1a-e6b9-40b9-96b2-3edcfeb8bad9", "content-length": "2", "date": "Tue, 25 Mar 2025 22:37:03 GMT"} })
58762025-03-25T22:37:04.315ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
58772025-03-25T22:37:04.315ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d5240e7-5e44-4e65-8961-4f29782ceadf", "content-length": "52", "date": "Tue, 25 Mar 2025 22:37:03 GMT"} })
58782025-03-25T22:37:04.315ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
58792025-03-25T22:37:04.315ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
58802025-03-25T22:37:04.411ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c5efa0f-29da-45dd-a9bd-16e508961924", "content-length": "691", "date": "Tue, 25 Mar 2025 22:37:04 GMT"} })
58812025-03-25T22:37:04.411ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
58822025-03-25T22:37:04.411ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
58832025-03-25T22:37:04.411ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
58842025-03-25T22:37:04.412ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 428.815041ms iteration = 18 status = {}
58852025-03-25T22:37:04.533ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "e422e053-db62-442d-b128-fa1892f46a85", "date": "Tue, 25 Mar 2025 22:37:04 GMT"} })
58862025-03-25T22:37:04.533ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
58872025-03-25T22:37:04.533ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
58882025-03-25T22:37:04.533ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "6547759b-8878-4b09-9fff-1f4ceaf64ddf", "content-length": "682", "date": "Tue, 25 Mar 2025 22:37:04 GMT"} })
58892025-03-25T22:37:04.533ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58902025-03-25T22:37:04.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58912025-03-25T22:37:04.537ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58922025-03-25T22:37:04.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58932025-03-25T22:37:04.539ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58942025-03-25T22:37:04.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58952025-03-25T22:37:04.541ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58962025-03-25T22:37:04.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58972025-03-25T22:37:04.548ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 564.287731ms iteration = 18 status = {}
58982025-03-25T22:37:04.729ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
58992025-03-25T22:37:04.729ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
59002025-03-25T22:37:04.759ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
59012025-03-25T22:37:04.759ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:33468
59022025-03-25T22:37:04.759ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:33468 req_id = fe835a51-a9a5-4905-82af-295f1d82cc3c roles = RoleSet { roles: {} } uri = /metrics/producers
59032025-03-25T22:37:04.760ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:33468 req_id = fe835a51-a9a5-4905-82af-295f1d82cc3c resource = Database result = Ok(()) uri = /metrics/producers
59042025-03-25T22:37:04.805ZINFOnexus: assigned collector to new producer collector_id = 26728504-e36d-4baa-856d-cd0ce7a172a2 file = nexus/src/app/oximeter.rs:76 producer_id = 56ee8e9a-ccae-49c2-ac33-94d5159567dd
59052025-03-25T22:37:04.805ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 45603 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:33468 req_id = fe835a51-a9a5-4905-82af-295f1d82cc3c response_code = 201 uri = /metrics/producers
59062025-03-25T22:37:04.805ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "fe835a51-a9a5-4905-82af-295f1d82cc3c", "content-length": "41", "date": "Tue, 25 Mar 2025 22:37:04 GMT"} })
59072025-03-25T22:37:04.807ZDEBGnexus (producer-registration-task): registered with nexus successfully
59082025-03-25T22:37:04.807ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
59092025-03-25T22:37:06.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 282 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 0593eb29-b38a-4fc2-b91a-b736729437ad response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
59102025-03-25T22:37:13.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 53 reason = Timeout
59112025-03-25T22:37:13.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59122025-03-25T22:37:13.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
59132025-03-25T22:37:13.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59142025-03-25T22:37:13.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59152025-03-25T22:37:13.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59162025-03-25T22:37:13.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59172025-03-25T22:37:13.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59182025-03-25T22:37:13.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59192025-03-25T22:37:13.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.80345ms iteration = 53 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
59202025-03-25T22:37:13.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
59212025-03-25T22:37:13.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59222025-03-25T22:37:13.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59232025-03-25T22:37:13.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59242025-03-25T22:37:13.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59252025-03-25T22:37:13.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59262025-03-25T22:37:13.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
59272025-03-25T22:37:13.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59282025-03-25T22:37:13.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59292025-03-25T22:37:14.001ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
59302025-03-25T22:37:14.001ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.064269ms iteration = 53 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
59312025-03-25T22:37:16.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 270 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 29e74652-8306-4b51-8a83-5d700a40eb74 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
59322025-03-25T22:37:23.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 54 reason = Timeout
59332025-03-25T22:37:23.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59342025-03-25T22:37:23.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 54 reason = Timeout
59352025-03-25T22:37:23.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59362025-03-25T22:37:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59372025-03-25T22:37:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59382025-03-25T22:37:23.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59392025-03-25T22:37:23.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59402025-03-25T22:37:23.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59412025-03-25T22:37:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
59422025-03-25T22:37:23.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59432025-03-25T22:37:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59442025-03-25T22:37:23.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59452025-03-25T22:37:23.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59462025-03-25T22:37:23.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59472025-03-25T22:37:23.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
59482025-03-25T22:37:23.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59492025-03-25T22:37:23.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59502025-03-25T22:37:24.002ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
59512025-03-25T22:37:24.002ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.296842ms iteration = 54 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
59522025-03-25T22:37:24.013ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 30.329524ms iteration = 54 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
59532025-03-25T22:37:26.085ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 455 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 5a1d934c-5e8b-40fd-98cc-3b9ac83d2c97 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
59542025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
59552025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 19 reason = Timeout
59562025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 10 reason = Timeout
59572025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
59582025-03-25T22:37:33.982ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:27:33.982241523 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
59592025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
59602025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 19 reason = Timeout
59612025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
59622025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
59632025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
59642025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
59652025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
59662025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
59672025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
59682025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
59692025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
59702025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
59712025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
59722025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
59732025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 19 reason = Timeout
59742025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
59752025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
59762025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
59772025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
59782025-03-25T22:37:33.982ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("09fed906-08a5-4e51-9e85-fba7211b9594.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4ba21a7b-c053-4a77-807f-34395131fce2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1367, tv_nsec: 363197696 } })
59792025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
59802025-03-25T22:37:33.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
59812025-03-25T22:37:33.982ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8f1103be-ae21-497e-8fb0-5373f3346d6f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1367, tv_nsec: 363296507 } })
59822025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 19 reason = Timeout
59832025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 10 reason = Timeout
59842025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 20 reason = Timeout
59852025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
59862025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
59872025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
59882025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 10 reason = Timeout
59892025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
59902025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 10 reason = Timeout
59912025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
59922025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 19 reason = Timeout
59932025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
59942025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 5 reason = Timeout
59952025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 55 reason = Timeout
59962025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
59972025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59982025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 55 reason = Timeout
59992025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60002025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 19 reason = Timeout
60012025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 19 reason = Timeout
60022025-03-25T22:37:33.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
60032025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
60042025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 19 reason = Timeout
60052025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60062025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
60072025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 19 reason = Timeout
60082025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 19 reason = Timeout
60092025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
60102025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
60112025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 10 reason = Timeout
60122025-03-25T22:37:33.983ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
60132025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
60142025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 19 reason = Timeout
60152025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
60162025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 19 reason = Timeout
60172025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 19 reason = Timeout
60182025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
60192025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
60202025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 19 reason = Timeout
60212025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
60222025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 19 reason = Timeout
60232025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
60242025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60252025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 11 reason = Timeout
60262025-03-25T22:37:33.983ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b
60272025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 19 reason = Timeout
60282025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 17.561\u{b5}s iteration = 11 status = {"enabled":false,"target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b"}
60292025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
60302025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
60312025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60322025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
60332025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
60342025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
60352025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60362025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
60372025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
60382025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
60392025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
60402025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
60412025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.309154ms iteration = 10 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
60422025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60432025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
60442025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.486807ms iteration = 10 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
60452025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60462025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
60472025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
60482025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60492025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
60502025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
60512025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60522025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
60532025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
60542025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60552025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
60562025-03-25T22:37:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
60572025-03-25T22:37:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
60582025-03-25T22:37:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
60592025-03-25T22:37:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
60602025-03-25T22:37:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60612025-03-25T22:37:33.984ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
60622025-03-25T22:37:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60632025-03-25T22:37:33.984ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
60642025-03-25T22:37:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60652025-03-25T22:37:33.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1367, tv_nsec: 365696161 } })
60662025-03-25T22:37:33.985ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
60672025-03-25T22:37:33.986ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.83298ms iteration = 19 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
60682025-03-25T22:37:33.988ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
60692025-03-25T22:37:33.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
60702025-03-25T22:37:33.989ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60712025-03-25T22:37:33.990ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:27:33.982241523 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
60722025-03-25T22:37:33.990ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 8.100368ms iteration = 10 status = {"expiration":"2025-03-25T22:27:33.982241523Z","pruned":[]}
60732025-03-25T22:37:33.991ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60742025-03-25T22:37:33.991ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
60752025-03-25T22:37:33.991ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
60762025-03-25T22:37:33.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
60772025-03-25T22:37:33.992ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1367, tv_nsec: 372560136 } })
60782025-03-25T22:37:33.992ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
60792025-03-25T22:37:33.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
60802025-03-25T22:37:33.993ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60812025-03-25T22:37:33.993ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 10.736615ms iteration = 19 status = {"errors":[],"garbage_collect_requested":[]}
60822025-03-25T22:37:33.993ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 10.729016ms iteration = 10 status = {"errors":[],"found_port_ok":[]}
60832025-03-25T22:37:33.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60842025-03-25T22:37:33.994ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60852025-03-25T22:37:33.995ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
60862025-03-25T22:37:33.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
60872025-03-25T22:37:33.996ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60882025-03-25T22:37:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
60892025-03-25T22:37:33.996ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
60902025-03-25T22:37:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60912025-03-25T22:37:33.997ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
60922025-03-25T22:37:33.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
60932025-03-25T22:37:33.997ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60942025-03-25T22:37:33.999ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60952025-03-25T22:37:33.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
60962025-03-25T22:37:33.999ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60972025-03-25T22:37:34.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60982025-03-25T22:37:34.002ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
60992025-03-25T22:37:34.002ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
61002025-03-25T22:37:34.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
61012025-03-25T22:37:34.005ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
61022025-03-25T22:37:34.009ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61032025-03-25T22:37:34.011ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61042025-03-25T22:37:34.012ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
61052025-03-25T22:37:34.012ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
61062025-03-25T22:37:34.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
61072025-03-25T22:37:34.013ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
61082025-03-25T22:37:34.013ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
61092025-03-25T22:37:34.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
61102025-03-25T22:37:34.015ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61112025-03-25T22:37:34.019ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
61122025-03-25T22:37:34.019ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61132025-03-25T22:37:34.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61142025-03-25T22:37:34.021ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61152025-03-25T22:37:34.022ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
61162025-03-25T22:37:34.022ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61172025-03-25T22:37:34.022ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
61182025-03-25T22:37:34.022ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
61192025-03-25T22:37:34.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
61202025-03-25T22:37:34.024ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
61212025-03-25T22:37:34.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61222025-03-25T22:37:34.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
61232025-03-25T22:37:34.024ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
61242025-03-25T22:37:34.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
61252025-03-25T22:37:34.025ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
61262025-03-25T22:37:34.025ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
61272025-03-25T22:37:34.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
61282025-03-25T22:37:34.027ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61292025-03-25T22:37:34.028ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
61302025-03-25T22:37:34.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
61312025-03-25T22:37:34.029ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
61322025-03-25T22:37:34.029ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
61332025-03-25T22:37:34.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
61342025-03-25T22:37:34.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
61352025-03-25T22:37:34.032ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
61362025-03-25T22:37:34.035ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
61372025-03-25T22:37:34.035ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61382025-03-25T22:37:34.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
61392025-03-25T22:37:34.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61402025-03-25T22:37:34.050ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
61412025-03-25T22:37:34.050ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
61422025-03-25T22:37:34.050ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
61432025-03-25T22:37:34.050ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 67.48241ms iteration = 10 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
61442025-03-25T22:37:34.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
61452025-03-25T22:37:34.052ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
61462025-03-25T22:37:34.052ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
61472025-03-25T22:37:34.058ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
61482025-03-25T22:37:34.058ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 75.460646ms iteration = 19 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
61492025-03-25T22:37:34.062ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61502025-03-25T22:37:34.062ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61512025-03-25T22:37:34.062ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1367, tv_nsec: 443421749 } })
61522025-03-25T22:37:34.063ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 80.05979ms iteration = 19 status = {"errors":[],"requests_created_ok":[]}
61532025-03-25T22:37:34.063ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
61542025-03-25T22:37:34.065ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 81.756641ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
61552025-03-25T22:37:34.066ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
61562025-03-25T22:37:34.066ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
61572025-03-25T22:37:34.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
61582025-03-25T22:37:34.067ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
61592025-03-25T22:37:34.067ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
61602025-03-25T22:37:34.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
61612025-03-25T22:37:34.070ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61622025-03-25T22:37:34.072ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
61632025-03-25T22:37:34.072ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
61642025-03-25T22:37:34.072ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
61652025-03-25T22:37:34.072ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
61662025-03-25T22:37:34.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
61672025-03-25T22:37:34.072ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
61682025-03-25T22:37:34.073ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61692025-03-25T22:37:34.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61702025-03-25T22:37:34.076ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 93.041167ms iteration = 19 status = {"errors":[],"finish_invoked_ok":[]}
61712025-03-25T22:37:34.077ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61722025-03-25T22:37:34.079ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
61732025-03-25T22:37:34.079ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
61742025-03-25T22:37:34.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
61752025-03-25T22:37:34.081ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61762025-03-25T22:37:34.084ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 101.688455ms iteration = 55 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
61772025-03-25T22:37:34.085ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
61782025-03-25T22:37:34.085ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
61792025-03-25T22:37:34.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
61802025-03-25T22:37:34.085ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61812025-03-25T22:37:34.086ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61822025-03-25T22:37:34.088ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
61832025-03-25T22:37:34.088ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
61842025-03-25T22:37:34.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
61852025-03-25T22:37:34.091ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 108.163802ms iteration = 19 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
61862025-03-25T22:37:34.091ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
61872025-03-25T22:37:34.091ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61882025-03-25T22:37:34.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61892025-03-25T22:37:34.096ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
61902025-03-25T22:37:34.097ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
61912025-03-25T22:37:34.098ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61922025-03-25T22:37:34.099ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61932025-03-25T22:37:34.099ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
61942025-03-25T22:37:34.100ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
61952025-03-25T22:37:34.103ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
61962025-03-25T22:37:34.103ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61972025-03-25T22:37:34.103ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
61982025-03-25T22:37:34.103ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
61992025-03-25T22:37:34.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
62002025-03-25T22:37:34.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
62012025-03-25T22:37:34.106ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 122.992133ms iteration = 5 status = {"nsuccess":5}
62022025-03-25T22:37:34.108ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
62032025-03-25T22:37:34.108ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
62042025-03-25T22:37:34.108ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
62052025-03-25T22:37:34.108ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
62062025-03-25T22:37:34.108ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62072025-03-25T22:37:34.109ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 125.707093ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
62082025-03-25T22:37:34.110ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62092025-03-25T22:37:34.112ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
62102025-03-25T22:37:34.114ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
62112025-03-25T22:37:34.114ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
62122025-03-25T22:37:34.115ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f945dfc-e08e-4f35-9cd0-1a94ac2843ee", "content-length": "26", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} })
62132025-03-25T22:37:34.115ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
62142025-03-25T22:37:34.115ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
62152025-03-25T22:37:34.115ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
62162025-03-25T22:37:34.115ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62172025-03-25T22:37:34.117ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62182025-03-25T22:37:34.117ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
62192025-03-25T22:37:34.120ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
62202025-03-25T22:37:34.120ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
62212025-03-25T22:37:34.120ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
62222025-03-25T22:37:34.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
62232025-03-25T22:37:34.123ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
62242025-03-25T22:37:34.124ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
62252025-03-25T22:37:34.124ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 140.804878ms iteration = 55 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
62262025-03-25T22:37:34.124ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
62272025-03-25T22:37:34.125ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
62282025-03-25T22:37:34.125ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
62292025-03-25T22:37:34.125ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 141.715174ms iteration = 10 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
62302025-03-25T22:37:34.126ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
62312025-03-25T22:37:34.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
62322025-03-25T22:37:34.127ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
62332025-03-25T22:37:34.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
62342025-03-25T22:37:34.146ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
62352025-03-25T22:37:34.146ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
62362025-03-25T22:37:34.146ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
62372025-03-25T22:37:34.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
62382025-03-25T22:37:34.155ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
62392025-03-25T22:37:34.155ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 0 version = 2
62402025-03-25T22:37:34.155ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
62412025-03-25T22:37:34.156ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
62422025-03-25T22:37:34.167ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
62432025-03-25T22:37:34.167ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
62442025-03-25T22:37:34.167ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
62452025-03-25T22:37:34.167ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62462025-03-25T22:37:34.181ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
62472025-03-25T22:37:34.181ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 197.91926ms iteration = 10 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
62482025-03-25T22:37:34.181ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:28:31.652727Z, time_modified: 2025-03-25T22:28:31.652727Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }) }
62492025-03-25T22:37:34.181ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 199.015009ms iteration = 10 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]}},"default_endpoint":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
62502025-03-25T22:37:34.193ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
62512025-03-25T22:37:34.193ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
62522025-03-25T22:37:34.194ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "44325152-d1c0-4390-ac64-e2f4cf36afc3", "content-length": "26", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} })
62532025-03-25T22:37:34.194ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
62542025-03-25T22:37:34.195ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
62552025-03-25T22:37:34.224ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 241.79312ms iteration = 20 status = {"physical_disks_added":0}
62562025-03-25T22:37:34.225ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
62572025-03-25T22:37:34.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
62582025-03-25T22:37:34.238ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
62592025-03-25T22:37:34.238ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
62602025-03-25T22:37:34.238ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "27376b5c-37ca-472a-9e80-b1700019e1db", "content-length": "26", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} })
62612025-03-25T22:37:34.238ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
62622025-03-25T22:37:34.241ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
62632025-03-25T22:37:34.259ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
62642025-03-25T22:37:34.260ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "4618b593-ab6d-4a62-a1a5-2c30bb180b88", "content-length": "1", "date": "Tue, 25 Mar 2025 22:37:33 GMT"} })
62652025-03-25T22:37:34.262ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
62662025-03-25T22:37:34.262ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
62672025-03-25T22:37:34.277ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
62682025-03-25T22:37:34.297ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
62692025-03-25T22:37:34.297ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "7d2292ab-a6d3-47f2-b575-cb291f277f3c", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
62702025-03-25T22:37:34.298ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
62712025-03-25T22:37:34.299ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "8860f69b-dd69-41d8-a37f-e796e9c79f0b", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
62722025-03-25T22:37:34.299ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
62732025-03-25T22:37:34.301ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
62742025-03-25T22:37:34.301ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 319.640249ms iteration = 12 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
62752025-03-25T22:37:34.370ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
62762025-03-25T22:37:34.370ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
62772025-03-25T22:37:34.374ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "305ecab3-114b-4fc3-b38e-5ee2585d7923", "content-length": "372", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} })
62782025-03-25T22:37:34.375ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
62792025-03-25T22:37:34.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
62802025-03-25T22:37:34.383ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62812025-03-25T22:37:34.384ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62822025-03-25T22:37:34.384ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62832025-03-25T22:37:34.385ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62842025-03-25T22:37:34.388ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
62852025-03-25T22:37:34.388ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
62862025-03-25T22:37:34.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
62872025-03-25T22:37:34.390ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
62882025-03-25T22:37:34.390ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
62892025-03-25T22:37:34.390ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
62902025-03-25T22:37:34.390ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
62912025-03-25T22:37:34.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
62922025-03-25T22:37:34.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
62932025-03-25T22:37:34.391ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
62942025-03-25T22:37:34.391ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
62952025-03-25T22:37:34.391ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
62962025-03-25T22:37:34.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62972025-03-25T22:37:34.392ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "e1202e55-68e5-4600-8f34-105975824a2b", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
62982025-03-25T22:37:34.392ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
62992025-03-25T22:37:34.394ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 412.575705ms iteration = 19 status = null
63002025-03-25T22:37:34.395ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "5811c60a-6b95-49e1-b2ab-43481fda363f", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
63012025-03-25T22:37:34.395ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
63022025-03-25T22:37:34.397ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 414.281026ms iteration = 19 status = 0
63032025-03-25T22:37:34.399ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 417.532056ms iteration = 19 status = {}
63042025-03-25T22:37:34.400ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "69b45a11-ba7c-4e4e-8332-600f916191d2", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
63052025-03-25T22:37:34.400ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
63062025-03-25T22:37:34.400ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
63072025-03-25T22:37:34.400ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
63082025-03-25T22:37:34.400ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:29:06.674111 UTC dns_group = internal generation = 1
63092025-03-25T22:37:34.400ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 418.299379ms iteration = 10 status = {"generation":1}
63102025-03-25T22:37:34.402ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63112025-03-25T22:37:34.402ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
63122025-03-25T22:37:34.403ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 420.617182ms iteration = 12 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
63132025-03-25T22:37:34.404ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 1 version = 2
63142025-03-25T22:37:34.404ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
63152025-03-25T22:37:34.404ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:29:07.710706 UTC dns_group = external generation = 2
63162025-03-25T22:37:34.404ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 422.418214ms iteration = 10 status = {"generation":2}
63172025-03-25T22:37:34.405ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
63182025-03-25T22:37:34.405ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
63192025-03-25T22:37:34.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
63202025-03-25T22:37:34.440ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
63212025-03-25T22:37:34.440ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
63222025-03-25T22:37:34.440ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 456.841492ms iteration = 19 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
63232025-03-25T22:37:34.463ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
63242025-03-25T22:37:34.464ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 480.672037ms iteration = 19 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
63252025-03-25T22:37:34.468ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63262025-03-25T22:37:34.471ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
63272025-03-25T22:37:34.471ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
63282025-03-25T22:37:34.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
63292025-03-25T22:37:34.474ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
63302025-03-25T22:37:34.474ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
63312025-03-25T22:37:34.475ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "014e7465-c19d-45c9-80ff-497b70473070", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} })
63322025-03-25T22:37:34.475ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63332025-03-25T22:37:34.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63342025-03-25T22:37:34.477ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63352025-03-25T22:37:34.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63362025-03-25T22:37:34.485ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63372025-03-25T22:37:34.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63382025-03-25T22:37:34.489ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
63392025-03-25T22:37:34.489ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63402025-03-25T22:37:34.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63412025-03-25T22:37:34.490ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63422025-03-25T22:37:34.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63432025-03-25T22:37:34.530ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
63442025-03-25T22:37:34.531ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "27461000-ddad-47b9-9be0-16343512a918", "content-length": "691", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} })
63452025-03-25T22:37:34.531ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
63462025-03-25T22:37:34.531ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
63472025-03-25T22:37:34.531ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
63482025-03-25T22:37:34.534ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 550.805976ms iteration = 19 status = {}
63492025-03-25T22:37:34.534ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
63502025-03-25T22:37:34.535ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
63512025-03-25T22:37:34.535ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "bdd5ded1-2df6-4101-a076-6378c403fd05", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} })
63522025-03-25T22:37:34.537ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 554.543774ms iteration = 19 status = {}
63532025-03-25T22:37:34.545ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
63542025-03-25T22:37:34.545ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e135d55-2d81-4d8a-a964-618cd82f9508", "content-length": "134", "date": "Tue, 25 Mar 2025 22:37:33 GMT"} })
63552025-03-25T22:37:34.545ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
63562025-03-25T22:37:34.545ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
63572025-03-25T22:37:34.545ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
63582025-03-25T22:37:34.545ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
63592025-03-25T22:37:34.545ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
63602025-03-25T22:37:34.545ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
63612025-03-25T22:37:34.545ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
63622025-03-25T22:37:34.546ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
63632025-03-25T22:37:34.546ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
63642025-03-25T22:37:34.546ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
63652025-03-25T22:37:34.546ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
63662025-03-25T22:37:34.546ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63672025-03-25T22:37:34.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63682025-03-25T22:37:34.549ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
63692025-03-25T22:37:34.549ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "86c5fb15-52ce-4148-bdf5-643a066fe15e", "content-length": "2", "date": "Tue, 25 Mar 2025 22:37:33 GMT"} })
63702025-03-25T22:37:34.549ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
63712025-03-25T22:37:34.549ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "758e66fe-1da3-4100-bc2b-599f14f4498a", "content-length": "52", "date": "Tue, 25 Mar 2025 22:37:33 GMT"} })
63722025-03-25T22:37:34.549ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
63732025-03-25T22:37:34.549ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
63742025-03-25T22:37:34.677ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "aa095ebf-3289-4e97-8a5e-049755fc5ac8", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} })
63752025-03-25T22:37:34.677ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
63762025-03-25T22:37:34.677ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
63772025-03-25T22:37:34.677ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "b29249de-1963-413c-bbbc-4964d3b35131", "content-length": "682", "date": "Tue, 25 Mar 2025 22:37:34 GMT"} })
63782025-03-25T22:37:34.677ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63792025-03-25T22:37:34.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63802025-03-25T22:37:34.682ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63812025-03-25T22:37:34.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63822025-03-25T22:37:34.686ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63832025-03-25T22:37:34.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63842025-03-25T22:37:34.689ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63852025-03-25T22:37:34.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63862025-03-25T22:37:34.696ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 713.669106ms iteration = 19 status = {}
63872025-03-25T22:37:36.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 257 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = a129e918-0eb6-4b09-824a-2d703de628f4 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
63882025-03-25T22:37:43.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
63892025-03-25T22:37:43.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 56 reason = Timeout
63902025-03-25T22:37:43.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63912025-03-25T22:37:43.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63922025-03-25T22:37:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63932025-03-25T22:37:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63942025-03-25T22:37:43.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63952025-03-25T22:37:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
63962025-03-25T22:37:43.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63972025-03-25T22:37:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63982025-03-25T22:37:43.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63992025-03-25T22:37:43.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64002025-03-25T22:37:43.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64012025-03-25T22:37:43.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
64022025-03-25T22:37:43.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64032025-03-25T22:37:43.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64042025-03-25T22:37:44.004ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
64052025-03-25T22:37:44.004ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.524993ms iteration = 56 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
64062025-03-25T22:37:44.015ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
64072025-03-25T22:37:44.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
64082025-03-25T22:37:44.055ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 71.517455ms iteration = 56 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
64092025-03-25T22:37:46.079ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:49000
64102025-03-25T22:37:46.079ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49000 req_id = 0529f636-bf41-4b7a-80d6-dfc2f4b242f6 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
64112025-03-25T22:37:46.079ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49000 req_id = 0529f636-bf41-4b7a-80d6-dfc2f4b242f6 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
64122025-03-25T22:37:46.082ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3102 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49000 req_id = 0529f636-bf41-4b7a-80d6-dfc2f4b242f6 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
64132025-03-25T22:37:46.082ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49000 req_id = 5cf53252-db9d-4ff4-8efd-677160d21433 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
64142025-03-25T22:37:46.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49000 req_id = 5cf53252-db9d-4ff4-8efd-677160d21433 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
64152025-03-25T22:37:46.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 262 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = c01901ae-30d0-48b2-81bb-7d7a19f891f4 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
64162025-03-25T22:37:46.085ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2787 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49000 req_id = 5cf53252-db9d-4ff4-8efd-677160d21433 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
64172025-03-25T22:37:53.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 57 reason = Timeout
64182025-03-25T22:37:53.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 57 reason = Timeout
64192025-03-25T22:37:53.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
64202025-03-25T22:37:53.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64212025-03-25T22:37:53.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
64222025-03-25T22:37:53.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64232025-03-25T22:37:53.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64242025-03-25T22:37:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
64252025-03-25T22:37:53.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64262025-03-25T22:37:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64272025-03-25T22:37:53.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64282025-03-25T22:37:53.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64292025-03-25T22:37:53.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64302025-03-25T22:37:53.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
64312025-03-25T22:37:53.997ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64322025-03-25T22:37:53.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64332025-03-25T22:37:54.007ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
64342025-03-25T22:37:54.007ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.958677ms iteration = 57 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
64352025-03-25T22:37:54.020ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
64362025-03-25T22:37:54.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
64372025-03-25T22:37:54.050ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 66.906741ms iteration = 57 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
64382025-03-25T22:37:56.085ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 288 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = dfa260af-6b43-4a02-892c-64043466eed3 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
64392025-03-25T22:38:03.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
64402025-03-25T22:38:03.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
64412025-03-25T22:38:03.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
64422025-03-25T22:38:03.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
64432025-03-25T22:38:03.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
64442025-03-25T22:38:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
64452025-03-25T22:38:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
64462025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 20 reason = Timeout
64472025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
64482025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
64492025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
64502025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 20 reason = Timeout
64512025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64522025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
64532025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 20 reason = Timeout
64542025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
64552025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
64562025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 58 reason = Timeout
64572025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
64582025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 20 reason = Timeout
64592025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
64602025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 20 reason = Timeout
64612025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
64622025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64632025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 20 reason = Timeout
64642025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64652025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 20 reason = Timeout
64662025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
64672025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 21 reason = Timeout
64682025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 20 reason = Timeout
64692025-03-25T22:38:03.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
64702025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64712025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
64722025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 20 reason = Timeout
64732025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 20 reason = Timeout
64742025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
64752025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
64762025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 58 reason = Timeout
64772025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 20 reason = Timeout
64782025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64792025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
64802025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
64812025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64822025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
64832025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
64842025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
64852025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
64862025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64872025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
64882025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64892025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
64902025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64912025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
64922025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64932025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
64942025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
64952025-03-25T22:38:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
64962025-03-25T22:38:03.984ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1397, tv_nsec: 365262963 } })
64972025-03-25T22:38:03.985ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
64982025-03-25T22:38:03.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1397, tv_nsec: 365716951 } })
64992025-03-25T22:38:03.985ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
65002025-03-25T22:38:03.985ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65012025-03-25T22:38:03.987ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.94008ms iteration = 20 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
65022025-03-25T22:38:03.989ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
65032025-03-25T22:38:03.989ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65042025-03-25T22:38:03.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65052025-03-25T22:38:03.991ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65062025-03-25T22:38:03.991ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65072025-03-25T22:38:03.992ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65082025-03-25T22:38:03.994ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
65092025-03-25T22:38:03.994ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65102025-03-25T22:38:03.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65112025-03-25T22:38:03.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
65122025-03-25T22:38:03.995ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65132025-03-25T22:38:03.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65142025-03-25T22:38:03.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
65152025-03-25T22:38:03.996ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
65162025-03-25T22:38:03.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
65172025-03-25T22:38:04.011ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65182025-03-25T22:38:04.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65192025-03-25T22:38:04.012ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65202025-03-25T22:38:04.015ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
65212025-03-25T22:38:04.016ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
65222025-03-25T22:38:04.016ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1397, tv_nsec: 396546484 } })
65232025-03-25T22:38:04.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
65242025-03-25T22:38:04.016ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65252025-03-25T22:38:04.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65262025-03-25T22:38:04.016ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
65272025-03-25T22:38:04.019ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65282025-03-25T22:38:04.022ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
65292025-03-25T22:38:04.022ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
65302025-03-25T22:38:04.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
65312025-03-25T22:38:04.040ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
65322025-03-25T22:38:04.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
65332025-03-25T22:38:04.044ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65342025-03-25T22:38:04.046ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
65352025-03-25T22:38:04.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
65362025-03-25T22:38:04.047ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 64.299862ms iteration = 20 status = {"errors":[],"garbage_collect_requested":[]}
65372025-03-25T22:38:04.047ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 64.337243ms iteration = 20 status = {"errors":[],"finish_invoked_ok":[]}
65382025-03-25T22:38:04.047ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
65392025-03-25T22:38:04.047ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
65402025-03-25T22:38:04.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
65412025-03-25T22:38:04.049ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 66.096665ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
65422025-03-25T22:38:04.051ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 68.658413ms iteration = 58 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
65432025-03-25T22:38:04.055ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 72.251578ms iteration = 20 status = {"errors":[],"requests_created_ok":[]}
65442025-03-25T22:38:04.058ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
65452025-03-25T22:38:04.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
65462025-03-25T22:38:04.059ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65472025-03-25T22:38:04.061ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65482025-03-25T22:38:04.062ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
65492025-03-25T22:38:04.062ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65502025-03-25T22:38:04.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65512025-03-25T22:38:04.063ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65522025-03-25T22:38:04.064ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65532025-03-25T22:38:04.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65542025-03-25T22:38:04.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65552025-03-25T22:38:04.064ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
65562025-03-25T22:38:04.064ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
65572025-03-25T22:38:04.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
65582025-03-25T22:38:04.065ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
65592025-03-25T22:38:04.065ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
65602025-03-25T22:38:04.065ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
65612025-03-25T22:38:04.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
65622025-03-25T22:38:04.069ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65632025-03-25T22:38:04.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65642025-03-25T22:38:04.071ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
65652025-03-25T22:38:04.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
65662025-03-25T22:38:04.073ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
65672025-03-25T22:38:04.073ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
65682025-03-25T22:38:04.073ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
65692025-03-25T22:38:04.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
65702025-03-25T22:38:04.077ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65712025-03-25T22:38:04.077ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
65722025-03-25T22:38:04.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
65732025-03-25T22:38:04.079ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65742025-03-25T22:38:04.080ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
65752025-03-25T22:38:04.080ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65762025-03-25T22:38:04.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65772025-03-25T22:38:04.081ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
65782025-03-25T22:38:04.081ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
65792025-03-25T22:38:04.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
65802025-03-25T22:38:04.093ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
65812025-03-25T22:38:04.093ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
65822025-03-25T22:38:04.093ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5535962a-3015-4fb1-ab9d-35e002018216", "content-length": "26", "date": "Tue, 25 Mar 2025 22:38:04 GMT"} })
65832025-03-25T22:38:04.093ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
65842025-03-25T22:38:04.095ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
65852025-03-25T22:38:04.095ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
65862025-03-25T22:38:04.095ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
65872025-03-25T22:38:04.095ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "34a1d54f-2a93-48da-8c56-f92edc82ebe2", "content-length": "26", "date": "Tue, 25 Mar 2025 22:38:04 GMT"} })
65882025-03-25T22:38:04.095ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
65892025-03-25T22:38:04.096ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
65902025-03-25T22:38:04.096ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 113.287996ms iteration = 58 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
65912025-03-25T22:38:04.098ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
65922025-03-25T22:38:04.102ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 119.465269ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
65932025-03-25T22:38:04.106ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
65942025-03-25T22:38:04.106ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 123.458381ms iteration = 20 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
65952025-03-25T22:38:04.110ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
65962025-03-25T22:38:04.110ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
65972025-03-25T22:38:04.110ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3785ded0-789e-47a0-bc33-43d85774e2bd", "content-length": "26", "date": "Tue, 25 Mar 2025 22:38:04 GMT"} })
65982025-03-25T22:38:04.110ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
65992025-03-25T22:38:04.112ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
66002025-03-25T22:38:04.115ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66012025-03-25T22:38:04.119ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
66022025-03-25T22:38:04.119ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
66032025-03-25T22:38:04.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
66042025-03-25T22:38:04.121ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 138.431064ms iteration = 20 status = 0
66052025-03-25T22:38:04.123ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
66062025-03-25T22:38:04.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
66072025-03-25T22:38:04.123ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
66082025-03-25T22:38:04.123ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
66092025-03-25T22:38:04.123ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 140.415761ms iteration = 20 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
66102025-03-25T22:38:04.132ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
66112025-03-25T22:38:04.132ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 149.346604ms iteration = 20 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
66122025-03-25T22:38:04.145ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 161.858982ms iteration = 21 status = {"physical_disks_added":0}
66132025-03-25T22:38:04.155ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
66142025-03-25T22:38:04.155ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
66152025-03-25T22:38:04.159ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 176.279165ms iteration = 20 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
66162025-03-25T22:38:04.183ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
66172025-03-25T22:38:04.184ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fbbb274-ed6d-44f4-9c83-cbed9153fefd", "content-length": "1", "date": "Tue, 25 Mar 2025 22:38:03 GMT"} })
66182025-03-25T22:38:04.185ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
66192025-03-25T22:38:04.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
66202025-03-25T22:38:04.187ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 205.749403ms iteration = 20 status = null
66212025-03-25T22:38:04.187ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 205.878605ms iteration = 20 status = {}
66222025-03-25T22:38:04.192ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
66232025-03-25T22:38:04.192ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
66242025-03-25T22:38:04.200ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
66252025-03-25T22:38:04.200ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
66262025-03-25T22:38:04.200ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "a53bb382-289c-46bd-bdfb-ec0819c91b1f", "content-length": "372", "date": "Tue, 25 Mar 2025 22:38:04 GMT"} })
66272025-03-25T22:38:04.200ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66282025-03-25T22:38:04.201ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66292025-03-25T22:38:04.202ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66302025-03-25T22:38:04.205ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
66312025-03-25T22:38:04.205ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
66322025-03-25T22:38:04.205ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
66332025-03-25T22:38:04.207ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66342025-03-25T22:38:04.208ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
66352025-03-25T22:38:04.208ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
66362025-03-25T22:38:04.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
66372025-03-25T22:38:04.252ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
66382025-03-25T22:38:04.252ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d213201-1632-453b-876e-7890948b0af7", "content-length": "691", "date": "Tue, 25 Mar 2025 22:38:04 GMT"} })
66392025-03-25T22:38:04.252ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
66402025-03-25T22:38:04.252ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
66412025-03-25T22:38:04.252ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
66422025-03-25T22:38:04.254ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 270.712907ms iteration = 20 status = {}
66432025-03-25T22:38:04.266ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
66442025-03-25T22:38:04.266ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
66452025-03-25T22:38:04.266ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "8787d76b-504d-47d4-9907-051e087b7581", "date": "Tue, 25 Mar 2025 22:38:04 GMT"} })
66462025-03-25T22:38:04.266ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66472025-03-25T22:38:04.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66482025-03-25T22:38:04.268ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66492025-03-25T22:38:04.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66502025-03-25T22:38:04.272ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66512025-03-25T22:38:04.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66522025-03-25T22:38:04.274ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
66532025-03-25T22:38:04.274ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
66542025-03-25T22:38:04.274ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "025e0da2-3a22-4ecb-bf76-1509525c47ee", "date": "Tue, 25 Mar 2025 22:38:04 GMT"} })
66552025-03-25T22:38:04.276ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66562025-03-25T22:38:04.276ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 292.838771ms iteration = 20 status = {}
66572025-03-25T22:38:04.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66582025-03-25T22:38:04.277ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
66592025-03-25T22:38:04.277ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66602025-03-25T22:38:04.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66612025-03-25T22:38:04.287ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
66622025-03-25T22:38:04.288ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6392eeb-745a-4068-9f0d-c3944dbcc629", "content-length": "134", "date": "Tue, 25 Mar 2025 22:38:03 GMT"} })
66632025-03-25T22:38:04.288ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
66642025-03-25T22:38:04.288ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
66652025-03-25T22:38:04.288ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
66662025-03-25T22:38:04.288ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
66672025-03-25T22:38:04.288ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66682025-03-25T22:38:04.288ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66692025-03-25T22:38:04.288ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66702025-03-25T22:38:04.288ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
66712025-03-25T22:38:04.288ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66722025-03-25T22:38:04.288ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
66732025-03-25T22:38:04.288ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
66742025-03-25T22:38:04.288ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66752025-03-25T22:38:04.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66762025-03-25T22:38:04.291ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
66772025-03-25T22:38:04.291ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4bd8b2e-8241-4c60-a45a-7729285cd4f0", "content-length": "2", "date": "Tue, 25 Mar 2025 22:38:03 GMT"} })
66782025-03-25T22:38:04.291ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
66792025-03-25T22:38:04.291ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "faa0fea0-8a79-4d3a-8244-40e4c493e3f8", "content-length": "52", "date": "Tue, 25 Mar 2025 22:38:04 GMT"} })
66802025-03-25T22:38:04.291ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
66812025-03-25T22:38:04.291ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
66822025-03-25T22:38:04.435ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "51336ddc-282c-4a0f-989d-09bd66de6acd", "date": "Tue, 25 Mar 2025 22:38:04 GMT"} })
66832025-03-25T22:38:04.435ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
66842025-03-25T22:38:04.435ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
66852025-03-25T22:38:04.435ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cd262bc-c727-401a-ba7e-823e2236c564", "content-length": "682", "date": "Tue, 25 Mar 2025 22:38:04 GMT"} })
66862025-03-25T22:38:04.435ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66872025-03-25T22:38:04.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66882025-03-25T22:38:04.439ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66892025-03-25T22:38:04.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66902025-03-25T22:38:04.443ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66912025-03-25T22:38:04.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66922025-03-25T22:38:04.444ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66932025-03-25T22:38:04.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66942025-03-25T22:38:04.453ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 470.098663ms iteration = 20 status = {}
66952025-03-25T22:38:06.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 268 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 228214d3-9f7d-4954-b0f2-a858432622e5 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
66962025-03-25T22:38:13.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 59 reason = Timeout
66972025-03-25T22:38:13.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 59 reason = Timeout
66982025-03-25T22:38:13.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66992025-03-25T22:38:13.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67002025-03-25T22:38:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67012025-03-25T22:38:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67022025-03-25T22:38:13.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67032025-03-25T22:38:13.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67042025-03-25T22:38:13.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67052025-03-25T22:38:13.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.095054ms iteration = 59 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
67062025-03-25T22:38:13.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
67072025-03-25T22:38:13.987ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67082025-03-25T22:38:13.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67092025-03-25T22:38:13.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67102025-03-25T22:38:13.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67112025-03-25T22:38:13.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67122025-03-25T22:38:13.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
67132025-03-25T22:38:13.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67142025-03-25T22:38:13.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67152025-03-25T22:38:13.999ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
67162025-03-25T22:38:13.999ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.534044ms iteration = 59 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
67172025-03-25T22:38:16.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 288 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 028f6253-409f-46b0-8680-a89827f5c485 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
67182025-03-25T22:38:23.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 60 reason = Timeout
67192025-03-25T22:38:23.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 60 reason = Timeout
67202025-03-25T22:38:23.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67212025-03-25T22:38:23.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67222025-03-25T22:38:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67232025-03-25T22:38:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67242025-03-25T22:38:23.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67252025-03-25T22:38:23.986ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67262025-03-25T22:38:23.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67272025-03-25T22:38:23.988ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.570933ms iteration = 60 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
67282025-03-25T22:38:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
67292025-03-25T22:38:23.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67302025-03-25T22:38:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67312025-03-25T22:38:23.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67322025-03-25T22:38:23.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67332025-03-25T22:38:23.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67342025-03-25T22:38:23.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
67352025-03-25T22:38:23.997ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67362025-03-25T22:38:23.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67372025-03-25T22:38:24.005ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
67382025-03-25T22:38:24.005ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.220385ms iteration = 60 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
67392025-03-25T22:38:26.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 307 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 3e104fd6-859f-4328-b999-7d0cfc01d820 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
67402025-03-25T22:38:31.078ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:41526
67412025-03-25T22:38:31.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = 9da8d1e6-6fd8-49bb-9f41-906b7bfad871 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
67422025-03-25T22:38:31.078ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = 9da8d1e6-6fd8-49bb-9f41-906b7bfad871 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
67432025-03-25T22:38:31.080ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2581 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = 9da8d1e6-6fd8-49bb-9f41-906b7bfad871 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
67442025-03-25T22:38:31.081ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = b5cd6a3e-fd87-4471-ba64-5eb3d5201359 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
67452025-03-25T22:38:31.081ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = b5cd6a3e-fd87-4471-ba64-5eb3d5201359 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
67462025-03-25T22:38:31.083ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2470 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = b5cd6a3e-fd87-4471-ba64-5eb3d5201359 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
67472025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
67482025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
67492025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
67502025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
67512025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
67522025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
67532025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 11 reason = Timeout
67542025-03-25T22:38:33.982ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:28:33.982510957 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
67552025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 21 reason = Timeout
67562025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
67572025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
67582025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 21 reason = Timeout
67592025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 21 reason = Timeout
67602025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
67612025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
67622025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
67632025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
67642025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
67652025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
67662025-03-25T22:38:33.982ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
67672025-03-25T22:38:33.982ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8f1103be-ae21-497e-8fb0-5373f3346d6f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1427, tv_nsec: 363364559 } })
67682025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
67692025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
67702025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
67712025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 21 reason = Timeout
67722025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 21 reason = Timeout
67732025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
67742025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 11 reason = Timeout
67752025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
67762025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
67772025-03-25T22:38:33.983ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
67782025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 21 reason = Timeout
67792025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
67802025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 21 reason = Timeout
67812025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
67822025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 3 reason = Timeout
67832025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
67842025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 21 reason = Timeout
67852025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
67862025-03-25T22:38:33.983ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("09fed906-08a5-4e51-9e85-fba7211b9594.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4ba21a7b-c053-4a77-807f-34395131fce2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1427, tv_nsec: 363507501 } })
67872025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
67882025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 3 reason = Timeout
67892025-03-25T22:38:33.983ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
67902025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
67912025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 21 reason = Timeout
67922025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
67932025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
67942025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
67952025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 22 reason = Timeout
67962025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
67972025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 21 reason = Timeout
67982025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 21 reason = Timeout
67992025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
68002025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
68012025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
68022025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 11 reason = Timeout
68032025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
68042025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 61 reason = Timeout
68052025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
68062025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 11 reason = Timeout
68072025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
68082025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 21 reason = Timeout
68092025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 2 reason = Timeout
68102025-03-25T22:38:33.983ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
68112025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
68122025-03-25T22:38:33.983ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
68132025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 11 reason = Timeout
68142025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
68152025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 4 reason = Timeout
68162025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 21 reason = Timeout
68172025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
68182025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
68192025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 12 reason = Timeout
68202025-03-25T22:38:33.983ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b
68212025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 15.801\u{b5}s iteration = 12 status = {"enabled":false,"target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b"}
68222025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 21 reason = Timeout
68232025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 21 reason = Timeout
68242025-03-25T22:38:33.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
68252025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68262025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68272025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Timeout
68282025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68292025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 21 reason = Timeout
68302025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
68312025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
68322025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
68332025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 61 reason = Timeout
68342025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
68352025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
68362025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
68372025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68382025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
68392025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.174651ms iteration = 11 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
68402025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68412025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
68422025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
68432025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68442025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
68452025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
68462025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
68472025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
68482025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
68492025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
68502025-03-25T22:38:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
68512025-03-25T22:38:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
68522025-03-25T22:38:33.984ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
68532025-03-25T22:38:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
68542025-03-25T22:38:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
68552025-03-25T22:38:33.984ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
68562025-03-25T22:38:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.428316ms iteration = 11 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
68572025-03-25T22:38:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68582025-03-25T22:38:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68592025-03-25T22:38:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68602025-03-25T22:38:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68612025-03-25T22:38:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68622025-03-25T22:38:33.988ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
68632025-03-25T22:38:33.988ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1427, tv_nsec: 368626384 } })
68642025-03-25T22:38:33.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
68652025-03-25T22:38:33.988ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
68662025-03-25T22:38:33.989ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
68672025-03-25T22:38:33.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
68682025-03-25T22:38:33.991ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1427, tv_nsec: 371550488 } })
68692025-03-25T22:38:33.991ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
68702025-03-25T22:38:33.991ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 8.969154ms iteration = 21 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
68712025-03-25T22:38:33.991ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:28:33.982510957 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
68722025-03-25T22:38:33.991ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 9.272669ms iteration = 11 status = {"expiration":"2025-03-25T22:28:33.982510957Z","pruned":[]}
68732025-03-25T22:38:33.991ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68742025-03-25T22:38:33.993ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68752025-03-25T22:38:33.994ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68762025-03-25T22:38:33.994ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
68772025-03-25T22:38:33.994ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
68782025-03-25T22:38:33.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
68792025-03-25T22:38:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
68802025-03-25T22:38:33.996ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
68812025-03-25T22:38:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
68822025-03-25T22:38:33.997ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
68832025-03-25T22:38:33.997ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
68842025-03-25T22:38:33.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
68852025-03-25T22:38:34.000ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
68862025-03-25T22:38:34.000ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
68872025-03-25T22:38:34.000ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
68882025-03-25T22:38:34.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68892025-03-25T22:38:34.001ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 18.665171ms iteration = 21 status = {"errors":[],"requests_created_ok":[]}
68902025-03-25T22:38:34.002ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68912025-03-25T22:38:34.004ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 21.083824ms iteration = 11 status = {"errors":[],"found_port_ok":[]}
68922025-03-25T22:38:34.004ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
68932025-03-25T22:38:34.004ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68942025-03-25T22:38:34.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68952025-03-25T22:38:34.013ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
68962025-03-25T22:38:34.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
68972025-03-25T22:38:34.014ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68982025-03-25T22:38:34.017ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
68992025-03-25T22:38:34.017ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69002025-03-25T22:38:34.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69012025-03-25T22:38:34.020ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69022025-03-25T22:38:34.023ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69032025-03-25T22:38:34.023ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
69042025-03-25T22:38:34.023ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
69052025-03-25T22:38:34.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
69062025-03-25T22:38:34.027ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
69072025-03-25T22:38:34.027ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
69082025-03-25T22:38:34.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
69092025-03-25T22:38:34.030ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69102025-03-25T22:38:34.033ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69112025-03-25T22:38:34.033ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
69122025-03-25T22:38:34.033ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69132025-03-25T22:38:34.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69142025-03-25T22:38:34.036ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
69152025-03-25T22:38:34.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
69162025-03-25T22:38:34.037ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
69172025-03-25T22:38:34.038ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
69182025-03-25T22:38:34.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
69192025-03-25T22:38:34.039ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69202025-03-25T22:38:34.043ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
69212025-03-25T22:38:34.043ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
69222025-03-25T22:38:34.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
69232025-03-25T22:38:34.048ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
69242025-03-25T22:38:34.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
69252025-03-25T22:38:34.053ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1427, tv_nsec: 433859824 } })
69262025-03-25T22:38:34.053ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
69272025-03-25T22:38:34.055ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
69282025-03-25T22:38:34.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
69292025-03-25T22:38:34.065ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
69302025-03-25T22:38:34.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
69312025-03-25T22:38:34.071ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
69322025-03-25T22:38:34.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
69332025-03-25T22:38:34.079ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
69342025-03-25T22:38:34.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
69352025-03-25T22:38:34.079ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69362025-03-25T22:38:34.080ZDEBGnexus (BackgroundTasks): inventory_prune_one: eligible for removal background_task = inventory_collection candidates = [(0f881e28-6706-46d2-8a78-7e2e8704264b, 0), (087322d5-88cd-4d29-bb18-166fa9d2d9ae, 0), (f3a33071-613d-4c54-9027-e7e2e746ba30, 0), (f4145cfd-e21a-49bd-bfc2-974c4a454277, 0)] collection_id = 0f881e28-6706-46d2-8a78-7e2e8704264b
69372025-03-25T22:38:34.080ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 97.555649ms iteration = 21 status = {"errors":[],"garbage_collect_requested":[]}
69382025-03-25T22:38:34.080ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69392025-03-25T22:38:34.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69402025-03-25T22:38:34.081ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
69412025-03-25T22:38:34.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
69422025-03-25T22:38:34.083ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 100.023925ms iteration = 21 status = {"errors":[],"finish_invoked_ok":[]}
69432025-03-25T22:38:34.083ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
69442025-03-25T22:38:34.083ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
69452025-03-25T22:38:34.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
69462025-03-25T22:38:34.088ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69472025-03-25T22:38:34.088ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
69482025-03-25T22:38:34.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
69492025-03-25T22:38:34.090ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
69502025-03-25T22:38:34.090ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
69512025-03-25T22:38:34.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
69522025-03-25T22:38:34.091ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69532025-03-25T22:38:34.092ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
69542025-03-25T22:38:34.092ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 108.862756ms iteration = 2 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
69552025-03-25T22:38:34.093ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
69562025-03-25T22:38:34.093ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
69572025-03-25T22:38:34.093ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 110.023846ms iteration = 11 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
69582025-03-25T22:38:34.093ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 110.697099ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
69592025-03-25T22:38:34.095ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
69602025-03-25T22:38:34.095ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
69612025-03-25T22:38:34.095ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69622025-03-25T22:38:34.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
69632025-03-25T22:38:34.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69642025-03-25T22:38:34.096ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69652025-03-25T22:38:34.098ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
69662025-03-25T22:38:34.099ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
69672025-03-25T22:38:34.099ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
69682025-03-25T22:38:34.099ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
69692025-03-25T22:38:34.099ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
69702025-03-25T22:38:34.099ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
69712025-03-25T22:38:34.099ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
69722025-03-25T22:38:34.100ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
69732025-03-25T22:38:34.100ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69742025-03-25T22:38:34.100ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69752025-03-25T22:38:34.100ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69762025-03-25T22:38:34.102ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69772025-03-25T22:38:34.104ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
69782025-03-25T22:38:34.104ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
69792025-03-25T22:38:34.106ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
69802025-03-25T22:38:34.106ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
69812025-03-25T22:38:34.106ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
69822025-03-25T22:38:34.106ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 123.325479ms iteration = 11 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
69832025-03-25T22:38:34.106ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
69842025-03-25T22:38:34.106ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69852025-03-25T22:38:34.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
69862025-03-25T22:38:34.108ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
69872025-03-25T22:38:34.116ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69882025-03-25T22:38:34.116ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69892025-03-25T22:38:34.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69902025-03-25T22:38:34.117ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
69912025-03-25T22:38:34.117ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
69922025-03-25T22:38:34.117ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
69932025-03-25T22:38:34.117ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
69942025-03-25T22:38:34.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
69952025-03-25T22:38:34.117ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
69962025-03-25T22:38:34.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
69972025-03-25T22:38:34.119ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69982025-03-25T22:38:34.119ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69992025-03-25T22:38:34.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
70002025-03-25T22:38:34.122ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
70012025-03-25T22:38:34.122ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70022025-03-25T22:38:34.122ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70032025-03-25T22:38:34.123ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
70042025-03-25T22:38:34.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
70052025-03-25T22:38:34.124ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
70062025-03-25T22:38:34.124ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
70072025-03-25T22:38:34.124ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
70082025-03-25T22:38:34.126ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70092025-03-25T22:38:34.126ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70102025-03-25T22:38:34.133ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
70112025-03-25T22:38:34.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
70122025-03-25T22:38:34.133ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70132025-03-25T22:38:34.133ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
70142025-03-25T22:38:34.133ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70152025-03-25T22:38:34.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70162025-03-25T22:38:34.133ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
70172025-03-25T22:38:34.133ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
70182025-03-25T22:38:34.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
70192025-03-25T22:38:34.135ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
70202025-03-25T22:38:34.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
70212025-03-25T22:38:34.135ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
70222025-03-25T22:38:34.135ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70232025-03-25T22:38:34.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70242025-03-25T22:38:34.143ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
70252025-03-25T22:38:34.147ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
70262025-03-25T22:38:34.147ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
70272025-03-25T22:38:34.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
70282025-03-25T22:38:34.154ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
70292025-03-25T22:38:34.154ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
70302025-03-25T22:38:34.154ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "21fefc07-afa3-4482-8ec3-f3fe5613e186", "content-length": "26", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
70312025-03-25T22:38:34.154ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
70322025-03-25T22:38:34.154ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
70332025-03-25T22:38:34.156ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
70342025-03-25T22:38:34.164ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70352025-03-25T22:38:34.165ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 181.805836ms iteration = 61 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
70362025-03-25T22:38:34.166ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
70372025-03-25T22:38:34.166ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 183.545448ms iteration = 21 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
70382025-03-25T22:38:34.168ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
70392025-03-25T22:38:34.169ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
70402025-03-25T22:38:34.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
70412025-03-25T22:38:34.173ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
70422025-03-25T22:38:34.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
70432025-03-25T22:38:34.177ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 194.183322ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
70442025-03-25T22:38:34.179ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 196.27599ms iteration = 21 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
70452025-03-25T22:38:34.233ZINFOnexus (BackgroundTasks): removed inventory collection background_task = inventory_collection collection_id = 0f881e28-6706-46d2-8a78-7e2e8704264b file = nexus/db-queries/src/db/datastore/inventory.rs:1442 ncabooses = 0 nclickhouse_keeper_membership = 0 ncollections = 0 ndatasets = 0 nerrors = 0 nnics = 0 nnvme_disk_firmware = 0 nphysical_disks = 0 nrot_pages = 0 nrots = 0 nsled_agent_zones = 0 nsled_agents = 0 nsps = 0 nzones = 0 nzpools = 0
70462025-03-25T22:38:34.233ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70472025-03-25T22:38:34.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70482025-03-25T22:38:34.241ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70492025-03-25T22:38:34.242ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70502025-03-25T22:38:34.257ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 274.760131ms iteration = 22 status = {"physical_disks_added":0}
70512025-03-25T22:38:34.261ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
70522025-03-25T22:38:34.261ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
70532025-03-25T22:38:34.261ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a279bfb-e8bd-40c1-947d-1d0a9d66fa5b", "content-length": "26", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
70542025-03-25T22:38:34.261ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
70552025-03-25T22:38:34.263ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
70562025-03-25T22:38:34.303ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
70572025-03-25T22:38:34.303ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
70582025-03-25T22:38:34.303ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fd9dc70-575a-4ff0-9cd2-17253e2d9621", "content-length": "372", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
70592025-03-25T22:38:34.303ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70602025-03-25T22:38:34.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70612025-03-25T22:38:34.304ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
70622025-03-25T22:38:34.304ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
70632025-03-25T22:38:34.304ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3e2eb49-c237-4957-9e6b-07cc0a3585a3", "content-length": "26", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
70642025-03-25T22:38:34.304ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
70652025-03-25T22:38:34.305ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70662025-03-25T22:38:34.306ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70672025-03-25T22:38:34.307ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
70682025-03-25T22:38:34.307ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
70692025-03-25T22:38:34.307ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
70702025-03-25T22:38:34.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
70712025-03-25T22:38:34.308ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
70722025-03-25T22:38:34.308ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
70732025-03-25T22:38:34.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
70742025-03-25T22:38:34.313ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
70752025-03-25T22:38:34.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
70762025-03-25T22:38:34.319ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70772025-03-25T22:38:34.324ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
70782025-03-25T22:38:34.324ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70792025-03-25T22:38:34.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70802025-03-25T22:38:34.327ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(087322d5-88cd-4d29-bb18-166fa9d2d9ae, 0), (f3a33071-613d-4c54-9027-e7e2e746ba30, 0), (f4145cfd-e21a-49bd-bfc2-974c4a454277, 0)]
70812025-03-25T22:38:34.327ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1427, tv_nsec: 707928123 } })
70822025-03-25T22:38:34.328ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
70832025-03-25T22:38:34.328ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 345.239906ms iteration = 61 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
70842025-03-25T22:38:34.334ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
70852025-03-25T22:38:34.360ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
70862025-03-25T22:38:34.360ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a154959-ba44-4b0a-8146-ef3e5301fabd", "content-length": "1", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
70872025-03-25T22:38:34.362ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
70882025-03-25T22:38:34.362ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 0 version = 2
70892025-03-25T22:38:34.362ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
70902025-03-25T22:38:34.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
70912025-03-25T22:38:34.363ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
70922025-03-25T22:38:34.363ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
70932025-03-25T22:38:34.363ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
70942025-03-25T22:38:34.363ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
70952025-03-25T22:38:34.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
70962025-03-25T22:38:34.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
70972025-03-25T22:38:34.364ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
70982025-03-25T22:38:34.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
70992025-03-25T22:38:34.365ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71002025-03-25T22:38:34.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71012025-03-25T22:38:34.367ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71022025-03-25T22:38:34.370ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
71032025-03-25T22:38:34.370ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
71042025-03-25T22:38:34.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
71052025-03-25T22:38:34.378ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71062025-03-25T22:38:34.381ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
71072025-03-25T22:38:34.381ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
71082025-03-25T22:38:34.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
71092025-03-25T22:38:34.385ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
71102025-03-25T22:38:34.385ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 403.228494ms iteration = 21 status = null
71112025-03-25T22:38:34.389ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "9c138eff-1028-40b0-be54-d9413d8cc7e5", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
71122025-03-25T22:38:34.389ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
71132025-03-25T22:38:34.391ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71142025-03-25T22:38:34.391ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
71152025-03-25T22:38:34.391ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
71162025-03-25T22:38:34.391ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 408.623963ms iteration = 21 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
71172025-03-25T22:38:34.392ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
71182025-03-25T22:38:34.392ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
71192025-03-25T22:38:34.392ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "779722cb-c2ee-479f-baa9-ffc39a569f79", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
71202025-03-25T22:38:34.392ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
71212025-03-25T22:38:34.392ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "d9dc6d32-11ee-43dd-95d3-9fef5bc65509", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
71222025-03-25T22:38:34.392ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
71232025-03-25T22:38:34.393ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
71242025-03-25T22:38:34.393ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 409.937607ms iteration = 21 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
71252025-03-25T22:38:34.393ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
71262025-03-25T22:38:34.394ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
71272025-03-25T22:38:34.394ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 412.27963ms iteration = 13 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
71282025-03-25T22:38:34.397ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
71292025-03-25T22:38:34.397ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
71302025-03-25T22:38:34.397ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71312025-03-25T22:38:34.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71322025-03-25T22:38:34.398ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
71332025-03-25T22:38:34.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
71342025-03-25T22:38:34.401ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
71352025-03-25T22:38:34.401ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
71362025-03-25T22:38:34.402ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
71372025-03-25T22:38:34.405ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 422.052037ms iteration = 21 status = 0
71382025-03-25T22:38:34.405ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
71392025-03-25T22:38:34.405ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
71402025-03-25T22:38:34.406ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
71412025-03-25T22:38:34.407ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
71422025-03-25T22:38:34.407ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71432025-03-25T22:38:34.410ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
71442025-03-25T22:38:34.410ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
71452025-03-25T22:38:34.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
71462025-03-25T22:38:34.411ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
71472025-03-25T22:38:34.411ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
71482025-03-25T22:38:34.412ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "7283083a-fe2b-4c1b-9751-8c83599bb7b2", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
71492025-03-25T22:38:34.412ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71502025-03-25T22:38:34.414ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 430.714875ms iteration = 21 status = {}
71512025-03-25T22:38:34.414ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
71522025-03-25T22:38:34.414ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 430.95642ms iteration = 11 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
71532025-03-25T22:38:34.415ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
71542025-03-25T22:38:34.415ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
71552025-03-25T22:38:34.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
71562025-03-25T22:38:34.418ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
71572025-03-25T22:38:34.418ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 435.522803ms iteration = 4 status = {"blueprint_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","inventory_collection_id":"f4145cfd-e21a-49bd-bfc2-974c4a454277","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
71582025-03-25T22:38:34.436ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
71592025-03-25T22:38:34.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
71602025-03-25T22:38:34.440ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71612025-03-25T22:38:34.440ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71622025-03-25T22:38:34.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71632025-03-25T22:38:34.444ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71642025-03-25T22:38:34.445ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71652025-03-25T22:38:34.445ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71662025-03-25T22:38:34.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
71672025-03-25T22:38:34.447ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
71682025-03-25T22:38:34.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
71692025-03-25T22:38:34.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
71702025-03-25T22:38:34.449ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71712025-03-25T22:38:34.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71722025-03-25T22:38:34.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
71732025-03-25T22:38:34.449ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
71742025-03-25T22:38:34.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
71752025-03-25T22:38:34.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
71762025-03-25T22:38:34.450ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
71772025-03-25T22:38:34.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
71782025-03-25T22:38:34.452ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 1 version = 2
71792025-03-25T22:38:34.452ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
71802025-03-25T22:38:34.452ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:29:07.710706 UTC dns_group = external generation = 2
71812025-03-25T22:38:34.452ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 469.486803ms iteration = 11 status = {"generation":2}
71822025-03-25T22:38:34.453ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
71832025-03-25T22:38:34.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
71842025-03-25T22:38:34.453ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71852025-03-25T22:38:34.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71862025-03-25T22:38:34.454ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71872025-03-25T22:38:34.454ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
71882025-03-25T22:38:34.454ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
71892025-03-25T22:38:34.454ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:29:06.674111 UTC dns_group = internal generation = 1
71902025-03-25T22:38:34.455ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 472.528228ms iteration = 11 status = {"generation":1}
71912025-03-25T22:38:34.456ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71922025-03-25T22:38:34.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71932025-03-25T22:38:34.457ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:28:31.652727Z, time_modified: 2025-03-25T22:28:31.652727Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }) }
71942025-03-25T22:38:34.457ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 474.601576ms iteration = 11 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]}},"default_endpoint":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
71952025-03-25T22:38:34.457ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
71962025-03-25T22:38:34.457ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
71972025-03-25T22:38:34.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
71982025-03-25T22:38:34.459ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71992025-03-25T22:38:34.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
72002025-03-25T22:38:34.460ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/artifacts-config
72012025-03-25T22:38:34.461ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts-config", status: 204, headers: {"x-request-id": "a282e030-f78b-4a61-8c3d-bf1f3501be82", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72022025-03-25T22:38:34.462ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
72032025-03-25T22:38:34.465ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
72042025-03-25T22:38:34.465ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "a279f811-e370-4305-bf84-0065ccee9cd7", "content-length": "26", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72052025-03-25T22:38:34.465ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:673 sled = http://[fd00:1122:3344:101::1]:12345
72062025-03-25T22:38:34.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
72072025-03-25T22:38:34.467ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 484.373894ms iteration = 3 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":2,"put_config_err":0,"put_config_ok":2,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"f7ac1017-36e7-40ec-9172-e4e1545e73c8","time":"2025-03-25T22:33:34.620334724Z"},{"operation":"list","target_sled":"f7ac1017-36e7-40ec-9172-e4e1545e73c8","time":"2025-03-25T22:33:34.623733676Z"},{"generation":1,"operation":"put_config","target_sled":"f7ac1017-36e7-40ec-9172-e4e1545e73c8","time":"2025-03-25T22:38:34.461896400Z"},{"operation":"list","target_sled":"f7ac1017-36e7-40ec-9172-e4e1545e73c8","time":"2025-03-25T22:38:34.465452755Z"}]}
72082025-03-25T22:38:34.469ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 486.764118ms iteration = 21 status = {}
72092025-03-25T22:38:34.501ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
72102025-03-25T22:38:34.501ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
72112025-03-25T22:38:34.501ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
72122025-03-25T22:38:34.501ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
72132025-03-25T22:38:34.501ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9009ac9-476a-42c5-8b1f-d98b93220163", "content-length": "476", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72142025-03-25T22:38:34.501ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
72152025-03-25T22:38:34.501ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "8df7db3c-6e2e-425f-bea3-40626073efa5", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
72162025-03-25T22:38:34.501ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
72172025-03-25T22:38:34.501ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6e6ef9d-a1c5-49d2-b186-7df657dcd103", "content-length": "734", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72182025-03-25T22:38:34.502ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
72192025-03-25T22:38:34.502ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2b5ddcb-030b-49e9-b48d-51452703d5b7", "content-length": "111", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72202025-03-25T22:38:34.502ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
72212025-03-25T22:38:34.502ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b927124-53e9-4710-9e85-bc474abff5dc", "content-length": "111", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72222025-03-25T22:38:34.502ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
72232025-03-25T22:38:34.503ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cba6daf-0196-4d8f-8d63-dd384684cbc8", "content-length": "167", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72242025-03-25T22:38:34.503ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "1178c97e-b73e-4d2c-bf57-3165afd156e5", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
72252025-03-25T22:38:34.503ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
72262025-03-25T22:38:34.503ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
72272025-03-25T22:38:34.503ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62c24b17-a1ec-4c95-82af-a5560a479593", "content-length": "167", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72282025-03-25T22:38:34.503ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
72292025-03-25T22:38:34.503ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10d60303-6282-4573-8a4a-3bc989b4c5c1", "content-length": "175", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72302025-03-25T22:38:34.503ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
72312025-03-25T22:38:34.504ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "92ecc92f-6d47-4852-957b-0cfb82c4620a", "content-length": "175", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72322025-03-25T22:38:34.504ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
72332025-03-25T22:38:34.504ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de900b6c-8520-43b2-bc2f-24b730f6ec26", "content-length": "702", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72342025-03-25T22:38:34.504ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
72352025-03-25T22:38:34.504ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b25c3805-339b-47b8-80ee-c4133f4e53d6", "content-length": "727", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72362025-03-25T22:38:34.504ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
72372025-03-25T22:38:34.504ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd804e20-4fb9-46ff-82b8-e0f98f9d061a", "content-length": "729", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72382025-03-25T22:38:34.504ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "d75f7d8c-2a6c-40c0-bbc6-3b1221c5e452", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
72392025-03-25T22:38:34.504ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
72402025-03-25T22:38:34.504ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
72412025-03-25T22:38:34.504ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa5e7427-c18e-4453-9297-3a82ead977de", "content-length": "728", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72422025-03-25T22:38:34.504ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
72432025-03-25T22:38:34.505ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8058877-9ae5-4397-a695-5901b7db8980", "content-length": "722", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72442025-03-25T22:38:34.505ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
72452025-03-25T22:38:34.505ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "437d39d6-868e-4a64-8eff-dbc9020020fa", "content-length": "109", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72462025-03-25T22:38:34.505ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
72472025-03-25T22:38:34.506ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df4e1d57-e0c8-4273-9227-3ecda04ec17d", "content-length": "109", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72482025-03-25T22:38:34.506ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
72492025-03-25T22:38:34.506ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "18de1618-ada3-4716-a4ce-57c4446f70c5", "content-length": "169", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72502025-03-25T22:38:34.506ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
72512025-03-25T22:38:34.506ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "790a98b4-276d-4b4a-91a1-43289bf30683", "content-length": "169", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72522025-03-25T22:38:34.506ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
72532025-03-25T22:38:34.506ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
72542025-03-25T22:38:34.506ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 524.392384ms iteration = 13 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
72552025-03-25T22:38:34.507ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "126a3541-e4e4-4951-a497-a6f9dd6cfe4f", "content-length": "178", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72562025-03-25T22:38:34.507ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
72572025-03-25T22:38:34.507ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c071e215-8ccf-42af-92d8-1c6ac51ff22f", "content-length": "178", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72582025-03-25T22:38:34.507ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
72592025-03-25T22:38:34.507ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ed2c95f-60df-41d4-b019-d1acedc812db", "content-length": "702", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72602025-03-25T22:38:34.507ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
72612025-03-25T22:38:34.507ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62bb3d66-6698-42fd-aa52-dbdb4a217066", "content-length": "727", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72622025-03-25T22:38:34.507ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
72632025-03-25T22:38:34.508ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a699cf53-1460-48cd-a3a8-5768705c81b5", "content-length": "729", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72642025-03-25T22:38:34.508ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
72652025-03-25T22:38:34.510ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "497af303-6056-42f2-b70f-1c54c9e31fc2", "content-length": "728", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72662025-03-25T22:38:34.511ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72672025-03-25T22:38:34.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72682025-03-25T22:38:34.514ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
72692025-03-25T22:38:34.515ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "611de750-bd37-4ab6-b715-20315c8333c7", "content-length": "691", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
72702025-03-25T22:38:34.515ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
72712025-03-25T22:38:34.515ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
72722025-03-25T22:38:34.515ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
72732025-03-25T22:38:34.517ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 534.382237ms iteration = 21 status = {}
72742025-03-25T22:38:34.521ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
72752025-03-25T22:38:34.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
72762025-03-25T22:38:34.552ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72772025-03-25T22:38:34.555ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72782025-03-25T22:38:34.555ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72792025-03-25T22:38:34.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72802025-03-25T22:38:34.557ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72812025-03-25T22:38:34.560ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72822025-03-25T22:38:34.560ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72832025-03-25T22:38:34.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72842025-03-25T22:38:34.562ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
72852025-03-25T22:38:34.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
72862025-03-25T22:38:34.564ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72872025-03-25T22:38:34.568ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
72882025-03-25T22:38:34.568ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
72892025-03-25T22:38:34.568ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
72902025-03-25T22:38:34.570ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
72912025-03-25T22:38:34.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
72922025-03-25T22:38:34.573ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
72932025-03-25T22:38:34.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
72942025-03-25T22:38:34.576ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
72952025-03-25T22:38:34.576ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
72962025-03-25T22:38:34.576ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
72972025-03-25T22:38:34.576ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
72982025-03-25T22:38:34.579ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
72992025-03-25T22:38:34.579ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
73002025-03-25T22:38:34.579ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73012025-03-25T22:38:34.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73022025-03-25T22:38:34.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73032025-03-25T22:38:34.581ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73042025-03-25T22:38:34.581ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73052025-03-25T22:38:34.582ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73062025-03-25T22:38:34.582ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73072025-03-25T22:38:34.584ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73082025-03-25T22:38:34.590ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
73092025-03-25T22:38:34.590ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73102025-03-25T22:38:34.591ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73112025-03-25T22:38:34.598ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
73122025-03-25T22:38:34.598ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c9a2eaa-238b-4701-a4fb-e055819d1180", "content-length": "134", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
73132025-03-25T22:38:34.598ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
73142025-03-25T22:38:34.598ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
73152025-03-25T22:38:34.599ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
73162025-03-25T22:38:34.599ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
73172025-03-25T22:38:34.599ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
73182025-03-25T22:38:34.599ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
73192025-03-25T22:38:34.599ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
73202025-03-25T22:38:34.599ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
73212025-03-25T22:38:34.599ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
73222025-03-25T22:38:34.599ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
73232025-03-25T22:38:34.599ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
73242025-03-25T22:38:34.599ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73252025-03-25T22:38:34.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73262025-03-25T22:38:34.602ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
73272025-03-25T22:38:34.602ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c2de172-de4b-4b28-8ab4-6974b43a181c", "content-length": "2", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
73282025-03-25T22:38:34.602ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
73292025-03-25T22:38:34.602ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "73ca778a-6768-43c1-97bd-a960dff8551c", "content-length": "52", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
73302025-03-25T22:38:34.602ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
73312025-03-25T22:38:34.602ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
73322025-03-25T22:38:34.628ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73332025-03-25T22:38:34.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73342025-03-25T22:38:34.630ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73352025-03-25T22:38:34.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73362025-03-25T22:38:34.631ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73372025-03-25T22:38:34.635ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
73382025-03-25T22:38:34.635ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73392025-03-25T22:38:34.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73402025-03-25T22:38:34.640ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73412025-03-25T22:38:34.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73422025-03-25T22:38:34.641ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73432025-03-25T22:38:34.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73442025-03-25T22:38:34.666ZDEBGnexus (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[fd00:1122:3344:101::1]:12345
73452025-03-25T22:38:34.669ZDEBGnexus (BackgroundTasks): client request SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/inventory
73462025-03-25T22:38:34.671ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73472025-03-25T22:38:34.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73482025-03-25T22:38:34.673ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73492025-03-25T22:38:34.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73502025-03-25T22:38:34.676ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73512025-03-25T22:38:34.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73522025-03-25T22:38:34.678ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73532025-03-25T22:38:34.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73542025-03-25T22:38:34.679ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73552025-03-25T22:38:34.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73562025-03-25T22:38:34.681ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73572025-03-25T22:38:34.685ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
73582025-03-25T22:38:34.685ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73592025-03-25T22:38:34.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73602025-03-25T22:38:34.686ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73612025-03-25T22:38:34.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73622025-03-25T22:38:34.687ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73632025-03-25T22:38:34.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73642025-03-25T22:38:34.689ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73652025-03-25T22:38:34.692ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
73662025-03-25T22:38:34.692ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73672025-03-25T22:38:34.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73682025-03-25T22:38:34.698ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 988adef7-d7d2-40f7-9c58-5a3b26ba2391, kind: Service { id: c8c65163-9425-400d-ac5c-65dfc41d0bf4 }, name: Name("external-dns-c8c65163-9425-400d-ac5c-65dfc41d0bf4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 164])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f88e38f4-26bc-4d44-8958-52780bb3169b, kind: Service { id: 8f1103be-ae21-497e-8fb0-5373f3346d6f }, name: Name("external-dns-8f1103be-ae21-497e-8fb0-5373f3346d6f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 24])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 1658d477-e9f8-4bb8-9566-907b5ca7268e, kind: Service { id: 22035ab1-4462-404d-8fdb-869e194ecaa9 }, name: Name("nexus-22035ab1-4462-404d-8fdb-869e194ecaa9"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 38])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7cb27bfc-265b-486b-9dfd-d94fd10b21da, kind: Service { id: 56ee8e9a-ccae-49c2-ac33-94d5159567dd }, name: Name("nexus-56ee8e9a-ccae-49c2-ac33-94d5159567dd"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 162])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9d699467-8090-45a8-b06c-f08cbb0184d7, kind: Service { id: 94680e6f-9c70-48a1-afd3-ee3593739739 }, name: Name("nexus-94680e6f-9c70-48a1-afd3-ee3593739739"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 189, 114])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
73692025-03-25T22:38:34.698ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73702025-03-25T22:38:34.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73712025-03-25T22:38:34.699ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73722025-03-25T22:38:34.701ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
73732025-03-25T22:38:34.701ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73742025-03-25T22:38:34.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73752025-03-25T22:38:34.703ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
73762025-03-25T22:38:34.703ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 988adef7-d7d2-40f7-9c58-5a3b26ba2391, kind: Service { id: c8c65163-9425-400d-ac5c-65dfc41d0bf4 }, name: Name("external-dns-c8c65163-9425-400d-ac5c-65dfc41d0bf4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 164])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f88e38f4-26bc-4d44-8958-52780bb3169b, kind: Service { id: 8f1103be-ae21-497e-8fb0-5373f3346d6f }, name: Name("external-dns-8f1103be-ae21-497e-8fb0-5373f3346d6f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 24])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 1658d477-e9f8-4bb8-9566-907b5ca7268e, kind: Service { id: 22035ab1-4462-404d-8fdb-869e194ecaa9 }, name: Name("nexus-22035ab1-4462-404d-8fdb-869e194ecaa9"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 38])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7cb27bfc-265b-486b-9dfd-d94fd10b21da, kind: Service { id: 56ee8e9a-ccae-49c2-ac33-94d5159567dd }, name: Name("nexus-56ee8e9a-ccae-49c2-ac33-94d5159567dd"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 162])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9d699467-8090-45a8-b06c-f08cbb0184d7, kind: Service { id: 94680e6f-9c70-48a1-afd3-ee3593739739 }, name: Name("nexus-94680e6f-9c70-48a1-afd3-ee3593739739"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 189, 114])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
73772025-03-25T22:38:34.703ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
73782025-03-25T22:38:34.714ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "b70bb14d-d051-4c86-8520-c832a161b516", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
73792025-03-25T22:38:34.714ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
73802025-03-25T22:38:34.714ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
73812025-03-25T22:38:34.715ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c93e09cc-e9c2-415d-82f6-b5c6e09501e3", "content-length": "682", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
73822025-03-25T22:38:34.715ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73832025-03-25T22:38:34.715ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73842025-03-25T22:38:34.719ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73852025-03-25T22:38:34.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73862025-03-25T22:38:34.723ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73872025-03-25T22:38:34.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73882025-03-25T22:38:34.725ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73892025-03-25T22:38:34.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73902025-03-25T22:38:34.733ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 749.970378ms iteration = 21 status = {}
73912025-03-25T22:38:34.830ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: f7ac1017-36e7-40ec-9172-e4e1545e73c8, time_created: 2025-03-25T22:29:06.944737Z, time_modified: 2025-03-25T22:29:06.944737Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
73922025-03-25T22:38:34.830ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
73932025-03-25T22:38:34.830ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73942025-03-25T22:38:34.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73952025-03-25T22:38:34.832ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
73962025-03-25T22:38:34.833ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
73972025-03-25T22:38:34.834ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73982025-03-25T22:38:34.839ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
73992025-03-25T22:38:34.871ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
74002025-03-25T22:38:34.875ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "c35ca176-31ce-41de-98ac-508c59358924", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
74012025-03-25T22:38:34.876ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
74022025-03-25T22:38:34.876ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 893.625558ms
74032025-03-25T22:38:34.876ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 893.633858ms iteration = 3 status = {}
74042025-03-25T22:38:35.121ZDEBGnexus (BackgroundTasks): client response SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "429e7b5d-0709-4d92-bdca-94ad460ba350", "content-length": "31973", "date": "Tue, 25 Mar 2025 22:38:34 GMT"} })
74052025-03-25T22:38:35.143ZDEBGnexus (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
74062025-03-25T22:38:35.143ZDEBGnexus (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
74072025-03-25T22:38:35.143ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
74082025-03-25T22:38:35.148ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74092025-03-25T22:38:35.148ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74102025-03-25T22:38:35.150ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74112025-03-25T22:38:35.152ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
74122025-03-25T22:38:35.159ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74132025-03-25T22:38:35.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74142025-03-25T22:38:35.338ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 4d968f43-baa1-4faf-a342-b36cbd45ae6a file = nexus/db-queries/src/db/datastore/inventory.rs:1010
74152025-03-25T22:38:35.338ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 4d968f43-baa1-4faf-a342-b36cbd45ae6a time_started = 2025-03-25 22:38:34.501208 UTC
74162025-03-25T22:38:35.338ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 5 reason = Dependency
74172025-03-25T22:38:35.338ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.35537404s iteration = 2 status = {"collection_id":"4d968f43-baa1-4faf-a342-b36cbd45ae6a","time_done":"2025-03-25 22:38:35.146138 UTC","time_started":"2025-03-25 22:38:34.501208 UTC"}
74182025-03-25T22:38:35.338ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 23 reason = Dependency
74192025-03-25T22:38:35.338ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
74202025-03-25T22:38:35.338ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
74212025-03-25T22:38:35.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
74222025-03-25T22:38:35.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
74232025-03-25T22:38:35.341ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74242025-03-25T22:38:35.344ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74252025-03-25T22:38:35.344ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
74262025-03-25T22:38:35.344ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
74272025-03-25T22:38:35.344ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
74282025-03-25T22:38:35.347ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
74292025-03-25T22:38:35.347ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
74302025-03-25T22:38:35.348ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
74312025-03-25T22:38:35.350ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
74322025-03-25T22:38:35.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
74332025-03-25T22:38:35.352ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74342025-03-25T22:38:35.354ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
74352025-03-25T22:38:35.355ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
74362025-03-25T22:38:35.355ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
74372025-03-25T22:38:35.456ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 118.217746ms iteration = 23 status = {"physical_disks_added":0}
74382025-03-25T22:38:35.536ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
74392025-03-25T22:38:35.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
74402025-03-25T22:38:35.567ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
74412025-03-25T22:38:35.568ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
74422025-03-25T22:38:35.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74432025-03-25T22:38:35.604ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
74442025-03-25T22:38:35.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
74452025-03-25T22:38:35.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
74462025-03-25T22:38:35.606ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74472025-03-25T22:38:35.608ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
74482025-03-25T22:38:35.608ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
74492025-03-25T22:38:35.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
74502025-03-25T22:38:35.610ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
74512025-03-25T22:38:35.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
74522025-03-25T22:38:35.611ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74532025-03-25T22:38:35.613ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
74542025-03-25T22:38:35.613ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
74552025-03-25T22:38:35.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
74562025-03-25T22:38:35.614ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74572025-03-25T22:38:35.616ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
74582025-03-25T22:38:35.616ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
74592025-03-25T22:38:35.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
74602025-03-25T22:38:35.618ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
74612025-03-25T22:38:35.619ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 280.343213ms iteration = 5 status = {"blueprint_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","inventory_collection_id":"d7fef609-14c1-4b73-92ea-06a3e93639cf","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
74622025-03-25T22:38:36.085ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 304 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 98b22004-acaf-4d1e-8eed-ca78231e8a44 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
74632025-03-25T22:38:43.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 62 reason = Timeout
74642025-03-25T22:38:43.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 62 reason = Timeout
74652025-03-25T22:38:43.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74662025-03-25T22:38:43.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74672025-03-25T22:38:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74682025-03-25T22:38:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74692025-03-25T22:38:43.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74702025-03-25T22:38:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74712025-03-25T22:38:43.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74722025-03-25T22:38:43.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74732025-03-25T22:38:43.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74742025-03-25T22:38:43.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74752025-03-25T22:38:43.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74762025-03-25T22:38:43.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
74772025-03-25T22:38:43.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74782025-03-25T22:38:43.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74792025-03-25T22:38:44.004ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
74802025-03-25T22:38:44.004ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.3741ms iteration = 62 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
74812025-03-25T22:38:44.014ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74822025-03-25T22:38:44.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74832025-03-25T22:38:44.046ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 63.161462ms iteration = 62 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
74842025-03-25T22:38:46.085ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 275 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = ee183e9c-6499-49d2-8e3c-81835bfbb75a response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
74852025-03-25T22:38:53.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 63 reason = Timeout
74862025-03-25T22:38:53.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 63 reason = Timeout
74872025-03-25T22:38:53.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74882025-03-25T22:38:53.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74892025-03-25T22:38:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74902025-03-25T22:38:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74912025-03-25T22:38:53.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74922025-03-25T22:38:53.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74932025-03-25T22:38:53.987ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74942025-03-25T22:38:53.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74952025-03-25T22:38:53.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74962025-03-25T22:38:53.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74972025-03-25T22:38:53.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74982025-03-25T22:38:53.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
74992025-03-25T22:38:53.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75002025-03-25T22:38:53.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75012025-03-25T22:38:54.003ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
75022025-03-25T22:38:54.003ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.217619ms iteration = 63 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
75032025-03-25T22:38:54.013ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
75042025-03-25T22:38:54.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
75052025-03-25T22:38:54.047ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.72052ms iteration = 63 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
75062025-03-25T22:38:56.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 366 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 7f5b9ae3-b502-4daa-ad4e-74c052a960f4 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
75072025-03-25T22:39:01.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = f66fd9c6-61ce-4494-b2e3-cc02d0daaa52 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
75082025-03-25T22:39:01.079ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = f66fd9c6-61ce-4494-b2e3-cc02d0daaa52 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
75092025-03-25T22:39:01.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2866 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = f66fd9c6-61ce-4494-b2e3-cc02d0daaa52 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
75102025-03-25T22:39:01.081ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = d696b648-bece-486e-b896-9f5a1a5978d6 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
75112025-03-25T22:39:01.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = d696b648-bece-486e-b896-9f5a1a5978d6 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
75122025-03-25T22:39:01.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2762 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = d696b648-bece-486e-b896-9f5a1a5978d6 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
75132025-03-25T22:39:03.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 22 reason = Timeout
75142025-03-25T22:39:03.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 22 reason = Timeout
75152025-03-25T22:39:03.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
75162025-03-25T22:39:03.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 22 reason = Timeout
75172025-03-25T22:39:03.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
75182025-03-25T22:39:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
75192025-03-25T22:39:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
75202025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 22 reason = Timeout
75212025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 22 reason = Timeout
75222025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
75232025-03-25T22:39:03.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
75242025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 22 reason = Timeout
75252025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75262025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
75272025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
75282025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 64 reason = Timeout
75292025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 22 reason = Timeout
75302025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 22 reason = Timeout
75312025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75322025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
75332025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
75342025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 24 reason = Timeout
75352025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75362025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 22 reason = Timeout
75372025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
75382025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 22 reason = Timeout
75392025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
75402025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 22 reason = Timeout
75412025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
75422025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 22 reason = Timeout
75432025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 22 reason = Timeout
75442025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75452025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 22 reason = Timeout
75462025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
75472025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75482025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 22 reason = Timeout
75492025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
75502025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 64 reason = Timeout
75512025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
75522025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 22 reason = Timeout
75532025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
75542025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
75552025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75562025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75572025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
75582025-03-25T22:39:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
75592025-03-25T22:39:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
75602025-03-25T22:39:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
75612025-03-25T22:39:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
75622025-03-25T22:39:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
75632025-03-25T22:39:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
75642025-03-25T22:39:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75652025-03-25T22:39:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75662025-03-25T22:39:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
75672025-03-25T22:39:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
75682025-03-25T22:39:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75692025-03-25T22:39:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
75702025-03-25T22:39:03.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1457, tv_nsec: 365950895 } })
75712025-03-25T22:39:03.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1457, tv_nsec: 366127929 } })
75722025-03-25T22:39:03.985ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75732025-03-25T22:39:03.985ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
75742025-03-25T22:39:03.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
75752025-03-25T22:39:03.986ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.597974ms iteration = 22 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
75762025-03-25T22:39:03.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75772025-03-25T22:39:03.988ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
75782025-03-25T22:39:03.989ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75792025-03-25T22:39:03.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75802025-03-25T22:39:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
75812025-03-25T22:39:03.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75822025-03-25T22:39:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75832025-03-25T22:39:03.995ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75842025-03-25T22:39:03.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
75852025-03-25T22:39:03.999ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75862025-03-25T22:39:03.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75872025-03-25T22:39:04.017ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75882025-03-25T22:39:04.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75892025-03-25T22:39:04.019ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75902025-03-25T22:39:04.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75912025-03-25T22:39:04.046ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
75922025-03-25T22:39:04.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
75932025-03-25T22:39:04.053ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75942025-03-25T22:39:04.054ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75952025-03-25T22:39:04.055ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
75962025-03-25T22:39:04.055ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
75972025-03-25T22:39:04.055ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
75982025-03-25T22:39:04.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
75992025-03-25T22:39:04.056ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
76002025-03-25T22:39:04.056ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
76012025-03-25T22:39:04.057ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
76022025-03-25T22:39:04.058ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
76032025-03-25T22:39:04.058ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76042025-03-25T22:39:04.058ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
76052025-03-25T22:39:04.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
76062025-03-25T22:39:04.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
76072025-03-25T22:39:04.059ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 75.961935ms iteration = 22 status = {"errors":[],"finish_invoked_ok":[]}
76082025-03-25T22:39:04.060ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
76092025-03-25T22:39:04.060ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
76102025-03-25T22:39:04.060ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
76112025-03-25T22:39:04.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
76122025-03-25T22:39:04.063ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1457, tv_nsec: 444037780 } })
76132025-03-25T22:39:04.064ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
76142025-03-25T22:39:04.064ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76152025-03-25T22:39:04.065ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76162025-03-25T22:39:04.065ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76172025-03-25T22:39:04.065ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 82.308611ms iteration = 22 status = {"errors":[],"garbage_collect_requested":[]}
76182025-03-25T22:39:04.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76192025-03-25T22:39:04.065ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
76202025-03-25T22:39:04.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
76212025-03-25T22:39:04.068ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
76222025-03-25T22:39:04.068ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
76232025-03-25T22:39:04.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
76242025-03-25T22:39:04.068ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
76252025-03-25T22:39:04.068ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
76262025-03-25T22:39:04.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
76272025-03-25T22:39:04.072ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
76282025-03-25T22:39:04.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
76292025-03-25T22:39:04.075ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76302025-03-25T22:39:04.076ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
76312025-03-25T22:39:04.076ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
76322025-03-25T22:39:04.077ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c62d490d-ee83-4e26-8091-51cc08fd04b4", "content-length": "26", "date": "Tue, 25 Mar 2025 22:39:04 GMT"} })
76332025-03-25T22:39:04.077ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
76342025-03-25T22:39:04.077ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76352025-03-25T22:39:04.078ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76362025-03-25T22:39:04.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76372025-03-25T22:39:04.078ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
76382025-03-25T22:39:04.078ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
76392025-03-25T22:39:04.078ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
76402025-03-25T22:39:04.078ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 95.422231ms iteration = 22 status = {"errors":[],"requests_created_ok":[]}
76412025-03-25T22:39:04.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
76422025-03-25T22:39:04.080ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
76432025-03-25T22:39:04.080ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76442025-03-25T22:39:04.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76452025-03-25T22:39:04.081ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76462025-03-25T22:39:04.082ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
76472025-03-25T22:39:04.083ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76482025-03-25T22:39:04.083ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
76492025-03-25T22:39:04.084ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
76502025-03-25T22:39:04.084ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76512025-03-25T22:39:04.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76522025-03-25T22:39:04.087ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
76532025-03-25T22:39:04.087ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76542025-03-25T22:39:04.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76552025-03-25T22:39:04.089ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 105.967922ms iteration = 22 status = 0
76562025-03-25T22:39:04.098ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 115.626789ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
76572025-03-25T22:39:04.100ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
76582025-03-25T22:39:04.100ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 117.170586ms iteration = 64 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
76592025-03-25T22:39:04.111ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
76602025-03-25T22:39:04.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
76612025-03-25T22:39:04.118ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 134.622165ms iteration = 22 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
76622025-03-25T22:39:04.120ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
76632025-03-25T22:39:04.120ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
76642025-03-25T22:39:04.120ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
76652025-03-25T22:39:04.120ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 137.187152ms iteration = 22 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
76662025-03-25T22:39:04.120ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "023155ad-ba81-41f1-a61d-d55d39c27442", "content-length": "26", "date": "Tue, 25 Mar 2025 22:39:04 GMT"} })
76672025-03-25T22:39:04.120ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
76682025-03-25T22:39:04.122ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
76692025-03-25T22:39:04.126ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 142.612951ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
76702025-03-25T22:39:04.126ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
76712025-03-25T22:39:04.126ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
76722025-03-25T22:39:04.126ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 142.756914ms iteration = 22 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
76732025-03-25T22:39:04.126ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
76742025-03-25T22:39:04.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
76752025-03-25T22:39:04.127ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76762025-03-25T22:39:04.130ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
76772025-03-25T22:39:04.130ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
76782025-03-25T22:39:04.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
76792025-03-25T22:39:04.136ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 153.249045ms iteration = 64 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
76802025-03-25T22:39:04.152ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
76812025-03-25T22:39:04.152ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
76822025-03-25T22:39:04.167ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
76832025-03-25T22:39:04.167ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 183.882984ms iteration = 22 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
76842025-03-25T22:39:04.200ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 216.534299ms iteration = 24 status = {"physical_disks_added":0}
76852025-03-25T22:39:04.229ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 247.365702ms iteration = 22 status = {}
76862025-03-25T22:39:04.242ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
76872025-03-25T22:39:04.242ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
76882025-03-25T22:39:04.242ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dba58e7-1ef3-4fd1-91fc-913f962042ef", "content-length": "26", "date": "Tue, 25 Mar 2025 22:39:04 GMT"} })
76892025-03-25T22:39:04.242ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
76902025-03-25T22:39:04.243ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
76912025-03-25T22:39:04.254ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
76922025-03-25T22:39:04.254ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
76932025-03-25T22:39:04.256ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76942025-03-25T22:39:04.258ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
76952025-03-25T22:39:04.258ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
76962025-03-25T22:39:04.258ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
76972025-03-25T22:39:04.260ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76982025-03-25T22:39:04.262ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
76992025-03-25T22:39:04.262ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
77002025-03-25T22:39:04.262ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
77012025-03-25T22:39:04.263ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
77022025-03-25T22:39:04.263ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bb556ad-be28-47e2-9933-13684b30b281", "content-length": "1", "date": "Tue, 25 Mar 2025 22:39:04 GMT"} })
77032025-03-25T22:39:04.265ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
77042025-03-25T22:39:04.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
77052025-03-25T22:39:04.269ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
77062025-03-25T22:39:04.269ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
77072025-03-25T22:39:04.270ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "60571eb5-575b-4331-9469-d605b17c67f6", "content-length": "372", "date": "Tue, 25 Mar 2025 22:39:04 GMT"} })
77082025-03-25T22:39:04.270ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77092025-03-25T22:39:04.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77102025-03-25T22:39:04.272ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 289.944978ms iteration = 22 status = null
77112025-03-25T22:39:04.318ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
77122025-03-25T22:39:04.318ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
77132025-03-25T22:39:04.378ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77142025-03-25T22:39:04.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77152025-03-25T22:39:04.381ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
77162025-03-25T22:39:04.381ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77172025-03-25T22:39:04.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77182025-03-25T22:39:04.395ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
77192025-03-25T22:39:04.396ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb19b38a-1fc5-4ecc-acee-5a0a8edd6064", "content-length": "134", "date": "Tue, 25 Mar 2025 22:39:04 GMT"} })
77202025-03-25T22:39:04.396ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
77212025-03-25T22:39:04.396ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
77222025-03-25T22:39:04.396ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
77232025-03-25T22:39:04.396ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
77242025-03-25T22:39:04.396ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77252025-03-25T22:39:04.396ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77262025-03-25T22:39:04.396ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77272025-03-25T22:39:04.396ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
77282025-03-25T22:39:04.396ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77292025-03-25T22:39:04.396ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
77302025-03-25T22:39:04.396ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
77312025-03-25T22:39:04.397ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77322025-03-25T22:39:04.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77332025-03-25T22:39:04.398ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
77342025-03-25T22:39:04.401ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
77352025-03-25T22:39:04.401ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "c99d8556-5f72-4bf5-95f7-fcd84fcf6885", "content-length": "2", "date": "Tue, 25 Mar 2025 22:39:04 GMT"} })
77362025-03-25T22:39:04.401ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
77372025-03-25T22:39:04.401ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "462b6a08-f90a-440b-80ac-2ad6cb527171", "content-length": "52", "date": "Tue, 25 Mar 2025 22:39:04 GMT"} })
77382025-03-25T22:39:04.401ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
77392025-03-25T22:39:04.401ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
77402025-03-25T22:39:04.445ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "62d964cb-b323-461b-b975-8ed3d3d3ef8c", "date": "Tue, 25 Mar 2025 22:39:04 GMT"} })
77412025-03-25T22:39:04.445ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77422025-03-25T22:39:04.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77432025-03-25T22:39:04.445ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "47a856c2-7c0a-4bb7-81db-87a6b77f7ded", "content-length": "691", "date": "Tue, 25 Mar 2025 22:39:04 GMT"} })
77442025-03-25T22:39:04.445ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
77452025-03-25T22:39:04.445ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
77462025-03-25T22:39:04.445ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
77472025-03-25T22:39:04.447ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 463.675386ms iteration = 22 status = {}
77482025-03-25T22:39:04.448ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77492025-03-25T22:39:04.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77502025-03-25T22:39:04.452ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77512025-03-25T22:39:04.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77522025-03-25T22:39:04.458ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
77532025-03-25T22:39:04.459ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
77542025-03-25T22:39:04.461ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "ae6c0f2f-74bd-486f-8bc3-43403fce0bde", "date": "Tue, 25 Mar 2025 22:39:04 GMT"} })
77552025-03-25T22:39:04.464ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 480.669476ms iteration = 22 status = {}
77562025-03-25T22:39:04.569ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "5bcdef99-0c83-4267-8e4d-f0f38ead4eac", "date": "Tue, 25 Mar 2025 22:39:04 GMT"} })
77572025-03-25T22:39:04.569ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
77582025-03-25T22:39:04.569ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
77592025-03-25T22:39:04.569ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6c45da5-a0a5-42b7-a1bb-974b17146824", "content-length": "682", "date": "Tue, 25 Mar 2025 22:39:04 GMT"} })
77602025-03-25T22:39:04.569ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77612025-03-25T22:39:04.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77622025-03-25T22:39:04.573ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77632025-03-25T22:39:04.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77642025-03-25T22:39:04.577ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77652025-03-25T22:39:04.577ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77662025-03-25T22:39:04.579ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77672025-03-25T22:39:04.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77682025-03-25T22:39:04.587ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 604.25085ms iteration = 22 status = {}
77692025-03-25T22:39:06.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 380 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 67de10da-1ece-4f16-91b2-2a4fade0d922 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
77702025-03-25T22:39:13.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 65 reason = Timeout
77712025-03-25T22:39:13.984ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 65 reason = Timeout
77722025-03-25T22:39:13.984ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77732025-03-25T22:39:13.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77742025-03-25T22:39:13.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77752025-03-25T22:39:13.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77762025-03-25T22:39:13.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77772025-03-25T22:39:13.986ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77782025-03-25T22:39:13.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77792025-03-25T22:39:13.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.340271ms iteration = 65 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
77802025-03-25T22:39:13.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
77812025-03-25T22:39:13.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77822025-03-25T22:39:13.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77832025-03-25T22:39:13.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77842025-03-25T22:39:13.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77852025-03-25T22:39:13.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77862025-03-25T22:39:13.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
77872025-03-25T22:39:13.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77882025-03-25T22:39:13.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77892025-03-25T22:39:14.000ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
77902025-03-25T22:39:14.000ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.42997ms iteration = 65 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
77912025-03-25T22:39:16.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 315 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 578a2e30-fefa-481e-92a6-33bc676d5c29 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
77922025-03-25T22:39:23.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 66 reason = Timeout
77932025-03-25T22:39:23.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 66 reason = Timeout
77942025-03-25T22:39:23.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77952025-03-25T22:39:23.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77962025-03-25T22:39:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77972025-03-25T22:39:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77982025-03-25T22:39:23.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77992025-03-25T22:39:23.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78002025-03-25T22:39:23.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
78012025-03-25T22:39:23.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.559754ms iteration = 66 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
78022025-03-25T22:39:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
78032025-03-25T22:39:23.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78042025-03-25T22:39:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78052025-03-25T22:39:23.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78062025-03-25T22:39:23.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78072025-03-25T22:39:23.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78082025-03-25T22:39:23.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
78092025-03-25T22:39:23.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78102025-03-25T22:39:23.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78112025-03-25T22:39:24.000ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
78122025-03-25T22:39:24.000ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.228305ms iteration = 66 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
78132025-03-25T22:39:26.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 291 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 2481af2a-9735-4d0b-b644-5c31533b76c4 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
78142025-03-25T22:39:31.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = 5348bd1f-d2e7-4ab3-ac98-49ed9a7774b4 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
78152025-03-25T22:39:31.079ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = 5348bd1f-d2e7-4ab3-ac98-49ed9a7774b4 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
78162025-03-25T22:39:31.082ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3459 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = 5348bd1f-d2e7-4ab3-ac98-49ed9a7774b4 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
78172025-03-25T22:39:31.082ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = cc0c1f60-82cc-47aa-97bf-83f3ef8add47 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
78182025-03-25T22:39:31.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = cc0c1f60-82cc-47aa-97bf-83f3ef8add47 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
78192025-03-25T22:39:31.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2118 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = cc0c1f60-82cc-47aa-97bf-83f3ef8add47 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
78202025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 12 reason = Timeout
78212025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 23 reason = Timeout
78222025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 14 reason = Timeout
78232025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
78242025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 12 reason = Timeout
78252025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 14 reason = Timeout
78262025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
78272025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 12 reason = Timeout
78282025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
78292025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
78302025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 23 reason = Timeout
78312025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 12 reason = Timeout
78322025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 12 reason = Timeout
78332025-03-25T22:39:33.982ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:29:33.982790001 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
78342025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
78352025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
78362025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
78372025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 12 reason = Timeout
78382025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 23 reason = Timeout
78392025-03-25T22:39:33.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
78402025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
78412025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
78422025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 23 reason = Timeout
78432025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 23 reason = Timeout
78442025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
78452025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78462025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 23 reason = Timeout
78472025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
78482025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 23 reason = Timeout
78492025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
78502025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
78512025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 67 reason = Timeout
78522025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
78532025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
78542025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 23 reason = Timeout
78552025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78562025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 23 reason = Timeout
78572025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 23 reason = Timeout
78582025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
78592025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
78602025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 23 reason = Timeout
78612025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 23 reason = Timeout
78622025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
78632025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 23 reason = Timeout
78642025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
78652025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
78662025-03-25T22:39:33.983ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("09fed906-08a5-4e51-9e85-fba7211b9594.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4ba21a7b-c053-4a77-807f-34395131fce2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1487, tv_nsec: 363679284 } })
78672025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 23 reason = Timeout
78682025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
78692025-03-25T22:39:33.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
78702025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 12 reason = Timeout
78712025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 12 reason = Timeout
78722025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
78732025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
78742025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
78752025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 23 reason = Timeout
78762025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 13 reason = Timeout
78772025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
78782025-03-25T22:39:33.983ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b
78792025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
78802025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 23 reason = Timeout
78812025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 17.1\u{b5}s iteration = 13 status = {"enabled":false,"target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b"}
78822025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
78832025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
78842025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 67 reason = Timeout
78852025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 12 reason = Timeout
78862025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 25 reason = Timeout
78872025-03-25T22:39:33.983ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
78882025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 12 reason = Timeout
78892025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78902025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
78912025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
78922025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
78932025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
78942025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78952025-03-25T22:39:33.983ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8f1103be-ae21-497e-8fb0-5373f3346d6f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1487, tv_nsec: 363926438 } })
78962025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78972025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
78982025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
78992025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
79002025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
79012025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
79022025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
79032025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
79042025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
79052025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
79062025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
79072025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
79082025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
79092025-03-25T22:39:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.291594ms iteration = 12 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
79102025-03-25T22:39:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
79112025-03-25T22:39:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79122025-03-25T22:39:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79132025-03-25T22:39:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
79142025-03-25T22:39:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
79152025-03-25T22:39:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
79162025-03-25T22:39:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
79172025-03-25T22:39:33.984ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
79182025-03-25T22:39:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
79192025-03-25T22:39:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.306624ms iteration = 12 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
79202025-03-25T22:39:33.984ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
79212025-03-25T22:39:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
79222025-03-25T22:39:33.985ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79232025-03-25T22:39:33.986ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.510934ms iteration = 23 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
79242025-03-25T22:39:33.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1487, tv_nsec: 366949464 } })
79252025-03-25T22:39:33.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
79262025-03-25T22:39:33.988ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1487, tv_nsec: 369077683 } })
79272025-03-25T22:39:33.988ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
79282025-03-25T22:39:33.988ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
79292025-03-25T22:39:33.989ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
79302025-03-25T22:39:33.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
79312025-03-25T22:39:33.991ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:29:33.982790001 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
79322025-03-25T22:39:33.991ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 8.754429ms iteration = 12 status = {"expiration":"2025-03-25T22:29:33.982790001Z","pruned":[]}
79332025-03-25T22:39:33.991ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79342025-03-25T22:39:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
79352025-03-25T22:39:33.996ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
79362025-03-25T22:39:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
79372025-03-25T22:39:34.001ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79382025-03-25T22:39:34.005ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
79392025-03-25T22:39:34.005ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
79402025-03-25T22:39:34.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
79412025-03-25T22:39:34.010ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79422025-03-25T22:39:34.011ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 28.168624ms iteration = 12 status = {"errors":[],"found_port_ok":[]}
79432025-03-25T22:39:34.011ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
79442025-03-25T22:39:34.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
79452025-03-25T22:39:34.013ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79462025-03-25T22:39:34.013ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
79472025-03-25T22:39:34.013ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
79482025-03-25T22:39:34.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
79492025-03-25T22:39:34.016ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79502025-03-25T22:39:34.018ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
79512025-03-25T22:39:34.018ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79522025-03-25T22:39:34.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79532025-03-25T22:39:34.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
79542025-03-25T22:39:34.021ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79552025-03-25T22:39:34.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79562025-03-25T22:39:34.022ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79572025-03-25T22:39:34.025ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
79582025-03-25T22:39:34.025ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
79592025-03-25T22:39:34.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
79602025-03-25T22:39:34.027ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79612025-03-25T22:39:34.030ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79622025-03-25T22:39:34.030ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
79632025-03-25T22:39:34.030ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
79642025-03-25T22:39:34.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
79652025-03-25T22:39:34.031ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79662025-03-25T22:39:34.033ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
79672025-03-25T22:39:34.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
79682025-03-25T22:39:34.034ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
79692025-03-25T22:39:34.035ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
79702025-03-25T22:39:34.043ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1487, tv_nsec: 423655028 } })
79712025-03-25T22:39:34.043ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79722025-03-25T22:39:34.043ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
79732025-03-25T22:39:34.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79742025-03-25T22:39:34.046ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79752025-03-25T22:39:34.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79762025-03-25T22:39:34.048ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79772025-03-25T22:39:34.051ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
79782025-03-25T22:39:34.051ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
79792025-03-25T22:39:34.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
79802025-03-25T22:39:34.054ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
79812025-03-25T22:39:34.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
79822025-03-25T22:39:34.057ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 74.220713ms iteration = 23 status = {"errors":[],"requests_created_ok":[]}
79832025-03-25T22:39:34.057ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79842025-03-25T22:39:34.057ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79852025-03-25T22:39:34.060ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79862025-03-25T22:39:34.061ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
79872025-03-25T22:39:34.061ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
79882025-03-25T22:39:34.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
79892025-03-25T22:39:34.063ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
79902025-03-25T22:39:34.063ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
79912025-03-25T22:39:34.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
79922025-03-25T22:39:34.064ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
79932025-03-25T22:39:34.064ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
79942025-03-25T22:39:34.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
79952025-03-25T22:39:34.066ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79962025-03-25T22:39:34.068ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
79972025-03-25T22:39:34.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
79982025-03-25T22:39:34.071ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
79992025-03-25T22:39:34.071ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
80002025-03-25T22:39:34.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
80012025-03-25T22:39:34.073ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80022025-03-25T22:39:34.075ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
80032025-03-25T22:39:34.075ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
80042025-03-25T22:39:34.075ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
80052025-03-25T22:39:34.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
80062025-03-25T22:39:34.076ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
80072025-03-25T22:39:34.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
80082025-03-25T22:39:34.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
80092025-03-25T22:39:34.078ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
80102025-03-25T22:39:34.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
80112025-03-25T22:39:34.080ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 96.991529ms iteration = 23 status = {"errors":[],"garbage_collect_requested":[]}
80122025-03-25T22:39:34.080ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 97.235334ms iteration = 23 status = {"errors":[],"finish_invoked_ok":[]}
80132025-03-25T22:39:34.082ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
80142025-03-25T22:39:34.082ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 99.105647ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
80152025-03-25T22:39:34.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
80162025-03-25T22:39:34.083ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80172025-03-25T22:39:34.087ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
80182025-03-25T22:39:34.087ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
80192025-03-25T22:39:34.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
80202025-03-25T22:39:34.092ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
80212025-03-25T22:39:34.092ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
80222025-03-25T22:39:34.092ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
80232025-03-25T22:39:34.092ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
80242025-03-25T22:39:34.092ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
80252025-03-25T22:39:34.092ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 109.402996ms iteration = 12 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
80262025-03-25T22:39:34.093ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
80272025-03-25T22:39:34.094ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 111.332141ms iteration = 25 status = {"physical_disks_added":0}
80282025-03-25T22:39:34.095ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
80292025-03-25T22:39:34.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
80302025-03-25T22:39:34.102ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
80312025-03-25T22:39:34.102ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
80322025-03-25T22:39:34.102ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
80332025-03-25T22:39:34.102ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
80342025-03-25T22:39:34.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
80352025-03-25T22:39:34.104ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
80362025-03-25T22:39:34.104ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
80372025-03-25T22:39:34.104ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
80382025-03-25T22:39:34.104ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
80392025-03-25T22:39:34.104ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
80402025-03-25T22:39:34.106ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
80412025-03-25T22:39:34.106ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 122.913432ms iteration = 23 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
80422025-03-25T22:39:34.107ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80432025-03-25T22:39:34.108ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
80442025-03-25T22:39:34.108ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 124.729435ms iteration = 67 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
80452025-03-25T22:39:34.110ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
80462025-03-25T22:39:34.110ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
80472025-03-25T22:39:34.110ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 127.163099ms iteration = 12 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
80482025-03-25T22:39:34.112ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
80492025-03-25T22:39:34.112ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
80502025-03-25T22:39:34.112ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
80512025-03-25T22:39:34.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
80522025-03-25T22:39:34.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
80532025-03-25T22:39:34.116ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
80542025-03-25T22:39:34.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
80552025-03-25T22:39:34.119ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80562025-03-25T22:39:34.119ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
80572025-03-25T22:39:34.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
80582025-03-25T22:39:34.123ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
80592025-03-25T22:39:34.123ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
80602025-03-25T22:39:34.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
80612025-03-25T22:39:34.127ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
80622025-03-25T22:39:34.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
80632025-03-25T22:39:34.131ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 147.997709ms iteration = 23 status = 0
80642025-03-25T22:39:34.140ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
80652025-03-25T22:39:34.140ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 157.227158ms iteration = 67 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
80662025-03-25T22:39:34.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
80672025-03-25T22:39:34.142ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80682025-03-25T22:39:34.144ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
80692025-03-25T22:39:34.144ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
80702025-03-25T22:39:34.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
80712025-03-25T22:39:34.148ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 165.081051ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
80722025-03-25T22:39:34.154ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
80732025-03-25T22:39:34.154ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
80742025-03-25T22:39:34.154ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fada054a-5167-4756-8508-e09b79e774e8", "content-length": "26", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} })
80752025-03-25T22:39:34.154ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
80762025-03-25T22:39:34.157ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
80772025-03-25T22:39:34.159ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
80782025-03-25T22:39:34.159ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
80792025-03-25T22:39:34.159ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
80802025-03-25T22:39:34.160ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
80812025-03-25T22:39:34.163ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
80822025-03-25T22:39:34.163ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
80832025-03-25T22:39:34.163ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
80842025-03-25T22:39:34.163ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80852025-03-25T22:39:34.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
80862025-03-25T22:39:34.165ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80872025-03-25T22:39:34.167ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
80882025-03-25T22:39:34.167ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
80892025-03-25T22:39:34.167ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f652e973-32eb-4f4e-a521-c8e26ceeff11", "content-length": "26", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} })
80902025-03-25T22:39:34.167ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
80912025-03-25T22:39:34.168ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
80922025-03-25T22:39:34.168ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
80932025-03-25T22:39:34.168ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80942025-03-25T22:39:34.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
80952025-03-25T22:39:34.169ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
80962025-03-25T22:39:34.169ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
80972025-03-25T22:39:34.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
80982025-03-25T22:39:34.169ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
80992025-03-25T22:39:34.170ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
81002025-03-25T22:39:34.172ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
81012025-03-25T22:39:34.172ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
81022025-03-25T22:39:34.172ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
81032025-03-25T22:39:34.172ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
81042025-03-25T22:39:34.172ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
81052025-03-25T22:39:34.172ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:29:06.674111 UTC dns_group = internal generation = 1
81062025-03-25T22:39:34.172ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 189.976705ms iteration = 12 status = {"generation":1}
81072025-03-25T22:39:34.188ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 205.565449ms iteration = 23 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
81082025-03-25T22:39:34.193ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:28:31.652727Z, time_modified: 2025-03-25T22:28:31.652727Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }) }
81092025-03-25T22:39:34.193ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 210.594411ms iteration = 12 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]}},"default_endpoint":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
81102025-03-25T22:39:34.207ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
81112025-03-25T22:39:34.207ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
81122025-03-25T22:39:34.207ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 224.612297ms iteration = 23 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
81132025-03-25T22:39:34.213ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
81142025-03-25T22:39:34.213ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 230.201089ms iteration = 23 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
81152025-03-25T22:39:34.223ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
81162025-03-25T22:39:34.223ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 0 version = 2
81172025-03-25T22:39:34.223ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
81182025-03-25T22:39:34.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
81192025-03-25T22:39:34.225ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81202025-03-25T22:39:34.230ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
81212025-03-25T22:39:34.230ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
81222025-03-25T22:39:34.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
81232025-03-25T22:39:34.257ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
81242025-03-25T22:39:34.257ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 273.987067ms iteration = 12 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
81252025-03-25T22:39:34.275ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
81262025-03-25T22:39:34.275ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
81272025-03-25T22:39:34.275ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e6c1b58-a451-4a31-964f-05488bd5bf3d", "content-length": "26", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} })
81282025-03-25T22:39:34.275ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
81292025-03-25T22:39:34.277ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
81302025-03-25T22:39:34.289ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
81312025-03-25T22:39:34.336ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 354.058158ms iteration = 23 status = {}
81322025-03-25T22:39:34.403ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
81332025-03-25T22:39:34.424ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
81342025-03-25T22:39:34.424ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d579dd8-8abd-4a3a-a38e-8e1c95540da8", "content-length": "691", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} })
81352025-03-25T22:39:34.424ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
81362025-03-25T22:39:34.424ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
81372025-03-25T22:39:34.424ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
81382025-03-25T22:39:34.427ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 444.305853ms iteration = 23 status = {}
81392025-03-25T22:39:34.431ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 1 version = 2
81402025-03-25T22:39:34.431ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
81412025-03-25T22:39:34.431ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:29:07.710706 UTC dns_group = external generation = 2
81422025-03-25T22:39:34.431ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 449.019559ms iteration = 12 status = {"generation":2}
81432025-03-25T22:39:34.486ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
81442025-03-25T22:39:34.486ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "6ac96d78-f3ee-4e85-9cbf-2a1846255068", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
81452025-03-25T22:39:34.486ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
81462025-03-25T22:39:34.488ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "26849661-93dd-4bab-8fbd-b30c82751b0a", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
81472025-03-25T22:39:34.488ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
81482025-03-25T22:39:34.490ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
81492025-03-25T22:39:34.490ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 508.29277ms iteration = 14 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
81502025-03-25T22:39:34.499ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
81512025-03-25T22:39:34.499ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
81522025-03-25T22:39:34.500ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bce6637-6049-498c-9db9-2f6eacbadc38", "content-length": "372", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} })
81532025-03-25T22:39:34.500ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81542025-03-25T22:39:34.500ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81552025-03-25T22:39:34.516ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
81562025-03-25T22:39:34.517ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "6877959f-57c4-47b9-9979-523d9570e225", "content-length": "1", "date": "Tue, 25 Mar 2025 22:39:33 GMT"} })
81572025-03-25T22:39:34.519ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
81582025-03-25T22:39:34.519ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
81592025-03-25T22:39:34.519ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "3ca92196-6afb-42ef-b067-0f7f7acc0c58", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} })
81602025-03-25T22:39:34.519ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81612025-03-25T22:39:34.519ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
81622025-03-25T22:39:34.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
81632025-03-25T22:39:34.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81642025-03-25T22:39:34.522ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81652025-03-25T22:39:34.522ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 539.937148ms iteration = 23 status = null
81662025-03-25T22:39:34.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81672025-03-25T22:39:34.525ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81682025-03-25T22:39:34.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81692025-03-25T22:39:34.526ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
81702025-03-25T22:39:34.526ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "d2918fbb-f138-401d-a214-3235e355e6cf", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
81712025-03-25T22:39:34.526ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
81722025-03-25T22:39:34.528ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "8dbb6901-b714-4dd4-91e8-dc32ab8f3f31", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
81732025-03-25T22:39:34.528ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
81742025-03-25T22:39:34.529ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
81752025-03-25T22:39:34.529ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
81762025-03-25T22:39:34.529ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "706f51ef-9dff-4f97-8cf2-6e47087d2a71", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
81772025-03-25T22:39:34.529ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
81782025-03-25T22:39:34.529ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "eee18bef-55ce-4274-aa84-9d6731fb42ad", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} })
81792025-03-25T22:39:34.532ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
81802025-03-25T22:39:34.532ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 549.568593ms iteration = 14 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
81812025-03-25T22:39:34.532ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 549.122545ms iteration = 23 status = {}
81822025-03-25T22:39:34.545ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81832025-03-25T22:39:34.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81842025-03-25T22:39:34.548ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
81852025-03-25T22:39:34.548ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81862025-03-25T22:39:34.548ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81872025-03-25T22:39:34.559ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
81882025-03-25T22:39:34.559ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2b10b1d-b7cd-4609-aaac-518f819d369e", "content-length": "134", "date": "Tue, 25 Mar 2025 22:39:33 GMT"} })
81892025-03-25T22:39:34.559ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
81902025-03-25T22:39:34.559ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
81912025-03-25T22:39:34.559ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
81922025-03-25T22:39:34.559ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
81932025-03-25T22:39:34.559ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81942025-03-25T22:39:34.559ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81952025-03-25T22:39:34.559ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81962025-03-25T22:39:34.560ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
81972025-03-25T22:39:34.560ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81982025-03-25T22:39:34.560ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
81992025-03-25T22:39:34.560ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
82002025-03-25T22:39:34.560ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82012025-03-25T22:39:34.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
82022025-03-25T22:39:34.562ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
82032025-03-25T22:39:34.562ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f6a4adb-666a-40cd-b9ca-3304fb86a09c", "content-length": "2", "date": "Tue, 25 Mar 2025 22:39:33 GMT"} })
82042025-03-25T22:39:34.562ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
82052025-03-25T22:39:34.562ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "363ab0fe-d7de-4670-83c9-2ea5a59ba5bd", "content-length": "52", "date": "Tue, 25 Mar 2025 22:39:33 GMT"} })
82062025-03-25T22:39:34.562ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
82072025-03-25T22:39:34.562ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
82082025-03-25T22:39:34.667ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "8f6787e9-2d72-4237-88be-c788f31b5cb8", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} })
82092025-03-25T22:39:34.667ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
82102025-03-25T22:39:34.667ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
82112025-03-25T22:39:34.672ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ba514ab-f5c0-409a-a090-fd6c0c285f4f", "content-length": "682", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} })
82122025-03-25T22:39:34.672ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82132025-03-25T22:39:34.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
82142025-03-25T22:39:34.676ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82152025-03-25T22:39:34.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
82162025-03-25T22:39:34.680ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82172025-03-25T22:39:34.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
82182025-03-25T22:39:34.682ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82192025-03-25T22:39:34.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
82202025-03-25T22:39:34.689ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 706.656298ms iteration = 23 status = {}
82212025-03-25T22:39:34.808ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
82222025-03-25T22:39:34.808ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
82232025-03-25T22:39:34.839ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
82242025-03-25T22:39:34.839ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:39737
82252025-03-25T22:39:34.839ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:39737 req_id = a6792e27-6e29-4c13-a41b-592aa77f5de4 roles = RoleSet { roles: {} } uri = /metrics/producers
82262025-03-25T22:39:34.839ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:39737 req_id = a6792e27-6e29-4c13-a41b-592aa77f5de4 resource = Database result = Ok(()) uri = /metrics/producers
82272025-03-25T22:39:34.881ZINFOnexus: assigned collector to new producer collector_id = 26728504-e36d-4baa-856d-cd0ce7a172a2 file = nexus/src/app/oximeter.rs:76 producer_id = 56ee8e9a-ccae-49c2-ac33-94d5159567dd
82282025-03-25T22:39:34.881ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 42222 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:39737 req_id = a6792e27-6e29-4c13-a41b-592aa77f5de4 response_code = 201 uri = /metrics/producers
82292025-03-25T22:39:34.881ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "a6792e27-6e29-4c13-a41b-592aa77f5de4", "content-length": "41", "date": "Tue, 25 Mar 2025 22:39:34 GMT"} })
82302025-03-25T22:39:34.884ZDEBGnexus (producer-registration-task): registered with nexus successfully
82312025-03-25T22:39:34.884ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
82322025-03-25T22:39:36.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 256 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 517fea78-ac19-47b8-9cdd-fcc867e2e42c response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
82332025-03-25T22:39:43.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 68 reason = Timeout
82342025-03-25T22:39:43.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 68 reason = Timeout
82352025-03-25T22:39:43.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82362025-03-25T22:39:43.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82372025-03-25T22:39:43.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82382025-03-25T22:39:43.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82392025-03-25T22:39:43.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82402025-03-25T22:39:43.986ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82412025-03-25T22:39:43.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82422025-03-25T22:39:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
82432025-03-25T22:39:43.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82442025-03-25T22:39:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82452025-03-25T22:39:43.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82462025-03-25T22:39:43.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82472025-03-25T22:39:43.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82482025-03-25T22:39:43.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
82492025-03-25T22:39:43.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82502025-03-25T22:39:43.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82512025-03-25T22:39:44.005ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
82522025-03-25T22:39:44.005ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.549393ms iteration = 68 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
82532025-03-25T22:39:44.021ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.440323ms iteration = 68 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
82542025-03-25T22:39:46.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 261 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 2b51f66a-83ae-4e8b-8f5c-83bc84d21c6a response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
82552025-03-25T22:39:53.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
82562025-03-25T22:39:53.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
82572025-03-25T22:39:53.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82582025-03-25T22:39:53.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82592025-03-25T22:39:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82602025-03-25T22:39:53.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82612025-03-25T22:39:53.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82622025-03-25T22:39:53.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82632025-03-25T22:39:53.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82642025-03-25T22:39:53.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.639067ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
82652025-03-25T22:39:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
82662025-03-25T22:39:53.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82672025-03-25T22:39:53.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82682025-03-25T22:39:53.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82692025-03-25T22:39:53.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82702025-03-25T22:39:53.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82712025-03-25T22:39:53.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
82722025-03-25T22:39:53.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82732025-03-25T22:39:53.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82742025-03-25T22:39:54.003ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
82752025-03-25T22:39:54.003ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.124297ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
82762025-03-25T22:39:56.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 338 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 0fb79026-217b-4a54-8344-8e8076c80836 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
82772025-03-25T22:40:01.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = 736697e6-eb0f-46c2-b7aa-4e32414a6816 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
82782025-03-25T22:40:01.078ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = 736697e6-eb0f-46c2-b7aa-4e32414a6816 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
82792025-03-25T22:40:01.080ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2481 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = 736697e6-eb0f-46c2-b7aa-4e32414a6816 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
82802025-03-25T22:40:01.081ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = cb5c7a87-50fb-4a02-8ba2-7ee1ed099824 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
82812025-03-25T22:40:01.081ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = cb5c7a87-50fb-4a02-8ba2-7ee1ed099824 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
82822025-03-25T22:40:01.087ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 6427 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:41526 req_id = cb5c7a87-50fb-4a02-8ba2-7ee1ed099824 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
82832025-03-25T22:40:03.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
82842025-03-25T22:40:03.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
82852025-03-25T22:40:03.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
82862025-03-25T22:40:03.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
82872025-03-25T22:40:03.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
82882025-03-25T22:40:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
82892025-03-25T22:40:03.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
82902025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
82912025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
82922025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
82932025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82942025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
82952025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
82962025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
82972025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82982025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
82992025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
83002025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
83012025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
83022025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
83032025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
83042025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
83052025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
83062025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 24 reason = Timeout
83072025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
83082025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
83092025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
83102025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
83112025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
83122025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
83132025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
83142025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 24 reason = Timeout
83152025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
83162025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
83172025-03-25T22:40:03.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
83182025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
83192025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83202025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
83212025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
83222025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83232025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83242025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83252025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
83262025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83272025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
83282025-03-25T22:40:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
83292025-03-25T22:40:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
83302025-03-25T22:40:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
83312025-03-25T22:40:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
83322025-03-25T22:40:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
83332025-03-25T22:40:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83342025-03-25T22:40:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
83352025-03-25T22:40:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
83362025-03-25T22:40:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83372025-03-25T22:40:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83382025-03-25T22:40:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
83392025-03-25T22:40:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
83402025-03-25T22:40:03.987ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1517, tv_nsec: 367395982 } })
83412025-03-25T22:40:03.987ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83422025-03-25T22:40:03.987ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
83432025-03-25T22:40:03.988ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1517, tv_nsec: 369130614 } })
83442025-03-25T22:40:03.988ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83452025-03-25T22:40:03.989ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
83462025-03-25T22:40:03.989ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
83472025-03-25T22:40:03.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
83482025-03-25T22:40:03.989ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
83492025-03-25T22:40:03.989ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83502025-03-25T22:40:03.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83512025-03-25T22:40:03.991ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 8.948743ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
83522025-03-25T22:40:03.991ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83532025-03-25T22:40:03.993ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
83542025-03-25T22:40:03.993ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
83552025-03-25T22:40:03.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83562025-03-25T22:40:03.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
83572025-03-25T22:40:03.995ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
83582025-03-25T22:40:03.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
83592025-03-25T22:40:03.999ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
83602025-03-25T22:40:04.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
83612025-03-25T22:40:04.000ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83622025-03-25T22:40:04.001ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83632025-03-25T22:40:04.003ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
83642025-03-25T22:40:04.003ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83652025-03-25T22:40:04.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83662025-03-25T22:40:04.004ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
83672025-03-25T22:40:04.004ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83682025-03-25T22:40:04.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83692025-03-25T22:40:04.005ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83702025-03-25T22:40:04.008ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83712025-03-25T22:40:04.008ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
83722025-03-25T22:40:04.008ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1517, tv_nsec: 389195439 } })
83732025-03-25T22:40:04.008ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
83742025-03-25T22:40:04.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
83752025-03-25T22:40:04.009ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
83762025-03-25T22:40:04.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83772025-03-25T22:40:04.009ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
83782025-03-25T22:40:04.009ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
83792025-03-25T22:40:04.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
83802025-03-25T22:40:04.011ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
83812025-03-25T22:40:04.011ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83822025-03-25T22:40:04.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83832025-03-25T22:40:04.016ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83842025-03-25T22:40:04.017ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83852025-03-25T22:40:04.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83862025-03-25T22:40:04.019ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
83872025-03-25T22:40:04.019ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83882025-03-25T22:40:04.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83892025-03-25T22:40:04.025ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83902025-03-25T22:40:04.025ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83912025-03-25T22:40:04.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83922025-03-25T22:40:04.030ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
83932025-03-25T22:40:04.030ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83942025-03-25T22:40:04.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83952025-03-25T22:40:04.034ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 51.211464ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
83962025-03-25T22:40:04.037ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83972025-03-25T22:40:04.043ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
83982025-03-25T22:40:04.043ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83992025-03-25T22:40:04.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
84002025-03-25T22:40:04.052ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
84012025-03-25T22:40:04.052ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 68.621462ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
84022025-03-25T22:40:04.052ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84032025-03-25T22:40:04.052ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84042025-03-25T22:40:04.053ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
84052025-03-25T22:40:04.053ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
84062025-03-25T22:40:04.053ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
84072025-03-25T22:40:04.053ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
84082025-03-25T22:40:04.056ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 72.665065ms iteration = 26 status = {"physical_disks_added":0}
84092025-03-25T22:40:04.060ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84102025-03-25T22:40:04.060ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
84112025-03-25T22:40:04.060ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
84122025-03-25T22:40:04.060ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
84132025-03-25T22:40:04.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84142025-03-25T22:40:04.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
84152025-03-25T22:40:04.062ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
84162025-03-25T22:40:04.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
84172025-03-25T22:40:04.062ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84182025-03-25T22:40:04.064ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 81.520547ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
84192025-03-25T22:40:04.066ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
84202025-03-25T22:40:04.066ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
84212025-03-25T22:40:04.066ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7ff800c-d4a9-4521-b8f8-a75aa0b58728", "content-length": "26", "date": "Tue, 25 Mar 2025 22:40:04 GMT"} })
84222025-03-25T22:40:04.066ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
84232025-03-25T22:40:04.066ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
84242025-03-25T22:40:04.066ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84252025-03-25T22:40:04.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84262025-03-25T22:40:04.068ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
84272025-03-25T22:40:04.069ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 86.100221ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
84282025-03-25T22:40:04.070ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 87.141409ms iteration = 24 status = 0
84292025-03-25T22:40:04.073ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
84302025-03-25T22:40:04.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
84312025-03-25T22:40:04.075ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
84322025-03-25T22:40:04.075ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 91.992467ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
84332025-03-25T22:40:04.078ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 94.504754ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
84342025-03-25T22:40:04.078ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
84352025-03-25T22:40:04.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
84362025-03-25T22:40:04.079ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 96.256566ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
84372025-03-25T22:40:04.085ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
84382025-03-25T22:40:04.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
84392025-03-25T22:40:04.088ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 104.6967ms iteration = 24 status = {"errors":[],"requests_created_ok":[]}
84402025-03-25T22:40:04.092ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
84412025-03-25T22:40:04.092ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
84422025-03-25T22:40:04.092ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "65a4c522-717e-4666-9d0f-54c480af6ad2", "content-length": "26", "date": "Tue, 25 Mar 2025 22:40:04 GMT"} })
84432025-03-25T22:40:04.092ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
84442025-03-25T22:40:04.094ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
84452025-03-25T22:40:04.098ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
84462025-03-25T22:40:04.098ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
84472025-03-25T22:40:04.098ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b75e3d8-c9c3-4f68-b531-2dc5ce08b291", "content-length": "26", "date": "Tue, 25 Mar 2025 22:40:04 GMT"} })
84482025-03-25T22:40:04.098ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
84492025-03-25T22:40:04.100ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
84502025-03-25T22:40:04.104ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
84512025-03-25T22:40:04.104ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
84522025-03-25T22:40:04.104ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 121.474325ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
84532025-03-25T22:40:04.115ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 132.172821ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
84542025-03-25T22:40:04.126ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
84552025-03-25T22:40:04.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
84562025-03-25T22:40:04.128ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84572025-03-25T22:40:04.130ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
84582025-03-25T22:40:04.130ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
84592025-03-25T22:40:04.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
84602025-03-25T22:40:04.131ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84612025-03-25T22:40:04.132ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84622025-03-25T22:40:04.133ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
84632025-03-25T22:40:04.133ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
84642025-03-25T22:40:04.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
84652025-03-25T22:40:04.134ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
84662025-03-25T22:40:04.134ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
84672025-03-25T22:40:04.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
84682025-03-25T22:40:04.163ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
84692025-03-25T22:40:04.163ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 180.347449ms iteration = 24 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
84702025-03-25T22:40:04.178ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 196.470134ms iteration = 24 status = {}
84712025-03-25T22:40:04.227ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
84722025-03-25T22:40:04.227ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4d491f0-4f03-4e0b-9984-7e94282d041b", "content-length": "1", "date": "Tue, 25 Mar 2025 22:40:03 GMT"} })
84732025-03-25T22:40:04.229ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
84742025-03-25T22:40:04.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
84752025-03-25T22:40:04.231ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 249.46802ms iteration = 24 status = null
84762025-03-25T22:40:04.239ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
84772025-03-25T22:40:04.239ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
84782025-03-25T22:40:04.239ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2b2904f-8fce-448b-97bb-ca831b5dbb9a", "content-length": "372", "date": "Tue, 25 Mar 2025 22:40:04 GMT"} })
84792025-03-25T22:40:04.239ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84802025-03-25T22:40:04.239ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84812025-03-25T22:40:04.250ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
84822025-03-25T22:40:04.250ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "13368049-db57-4caa-bbe4-ccf7f8010c75", "content-length": "691", "date": "Tue, 25 Mar 2025 22:40:04 GMT"} })
84832025-03-25T22:40:04.251ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
84842025-03-25T22:40:04.251ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
84852025-03-25T22:40:04.251ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
84862025-03-25T22:40:04.252ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 269.097957ms iteration = 24 status = {}
84872025-03-25T22:40:04.254ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
84882025-03-25T22:40:04.254ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
84892025-03-25T22:40:04.254ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "4e468a50-3cf9-496d-b27e-017d5591fa1b", "date": "Tue, 25 Mar 2025 22:40:04 GMT"} })
84902025-03-25T22:40:04.254ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84912025-03-25T22:40:04.254ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84922025-03-25T22:40:04.257ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84932025-03-25T22:40:04.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84942025-03-25T22:40:04.260ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84952025-03-25T22:40:04.260ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84962025-03-25T22:40:04.263ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
84972025-03-25T22:40:04.263ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
84982025-03-25T22:40:04.263ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "f31d86af-3ada-405b-9455-27b0cd943284", "date": "Tue, 25 Mar 2025 22:40:04 GMT"} })
84992025-03-25T22:40:04.265ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 282.042334ms iteration = 24 status = {}
85002025-03-25T22:40:04.300ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85012025-03-25T22:40:04.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85022025-03-25T22:40:04.302ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
85032025-03-25T22:40:04.302ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85042025-03-25T22:40:04.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85052025-03-25T22:40:04.313ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
85062025-03-25T22:40:04.313ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "20f6afad-dd51-4fee-9411-09703f1419fa", "content-length": "134", "date": "Tue, 25 Mar 2025 22:40:03 GMT"} })
85072025-03-25T22:40:04.313ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
85082025-03-25T22:40:04.313ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
85092025-03-25T22:40:04.313ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
85102025-03-25T22:40:04.313ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
85112025-03-25T22:40:04.313ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
85122025-03-25T22:40:04.313ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
85132025-03-25T22:40:04.313ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
85142025-03-25T22:40:04.313ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
85152025-03-25T22:40:04.313ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
85162025-03-25T22:40:04.313ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
85172025-03-25T22:40:04.314ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
85182025-03-25T22:40:04.314ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85192025-03-25T22:40:04.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85202025-03-25T22:40:04.315ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
85212025-03-25T22:40:04.316ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1143c84-b6e4-435a-9be1-10d077162b94", "content-length": "2", "date": "Tue, 25 Mar 2025 22:40:03 GMT"} })
85222025-03-25T22:40:04.316ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
85232025-03-25T22:40:04.316ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cd0068b-fb67-4ac4-afc1-4f1c4ba21463", "content-length": "52", "date": "Tue, 25 Mar 2025 22:40:03 GMT"} })
85242025-03-25T22:40:04.316ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
85252025-03-25T22:40:04.316ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
85262025-03-25T22:40:04.469ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "c23d2150-04d5-4eeb-95fb-abbd3f25d8c6", "date": "Tue, 25 Mar 2025 22:40:04 GMT"} })
85272025-03-25T22:40:04.469ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
85282025-03-25T22:40:04.469ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
85292025-03-25T22:40:04.475ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ca0a269-7108-4344-b1a4-d1aefa078c37", "content-length": "682", "date": "Tue, 25 Mar 2025 22:40:04 GMT"} })
85302025-03-25T22:40:04.475ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85312025-03-25T22:40:04.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85322025-03-25T22:40:04.479ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85332025-03-25T22:40:04.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85342025-03-25T22:40:04.483ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85352025-03-25T22:40:04.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85362025-03-25T22:40:04.485ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85372025-03-25T22:40:04.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85382025-03-25T22:40:04.494ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 510.861587ms iteration = 24 status = {}
85392025-03-25T22:40:06.085ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 298 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 94020c43-ee7c-4f02-a767-eb870eea34bd response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
85402025-03-25T22:40:13.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
85412025-03-25T22:40:13.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
85422025-03-25T22:40:13.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85432025-03-25T22:40:13.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85442025-03-25T22:40:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85452025-03-25T22:40:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85462025-03-25T22:40:13.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85472025-03-25T22:40:13.986ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85482025-03-25T22:40:13.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85492025-03-25T22:40:13.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.044394ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
85502025-03-25T22:40:13.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
85512025-03-25T22:40:13.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85522025-03-25T22:40:13.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85532025-03-25T22:40:13.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85542025-03-25T22:40:13.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85552025-03-25T22:40:13.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85562025-03-25T22:40:13.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
85572025-03-25T22:40:13.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85582025-03-25T22:40:13.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85592025-03-25T22:40:14.000ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
85602025-03-25T22:40:14.000ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.380317ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
85612025-03-25T22:40:16.085ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 314 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = c624830a-1e06-424d-9386-8a79ef7f02b5 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
85622025-03-25T22:40:23.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 72 reason = Timeout
85632025-03-25T22:40:23.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 72 reason = Timeout
85642025-03-25T22:40:23.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85652025-03-25T22:40:23.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85662025-03-25T22:40:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85672025-03-25T22:40:23.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85682025-03-25T22:40:23.985ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85692025-03-25T22:40:23.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85702025-03-25T22:40:23.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85712025-03-25T22:40:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
85722025-03-25T22:40:23.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85732025-03-25T22:40:23.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85742025-03-25T22:40:23.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85752025-03-25T22:40:23.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85762025-03-25T22:40:23.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85772025-03-25T22:40:23.994ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
85782025-03-25T22:40:23.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85792025-03-25T22:40:23.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85802025-03-25T22:40:24.004ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
85812025-03-25T22:40:24.004ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.499674ms iteration = 72 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
85822025-03-25T22:40:24.015ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.35852ms iteration = 72 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
85832025-03-25T22:40:26.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 273 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 266b032b-b301-4195-b8ea-4fd63832c39e response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
85842025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 13 reason = Timeout
85852025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 25 reason = Timeout
85862025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
85872025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 13 reason = Timeout
85882025-03-25T22:40:33.982ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:30:33.982631398 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
85892025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 13 reason = Timeout
85902025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
85912025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
85922025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 25 reason = Timeout
85932025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
85942025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 15 reason = Timeout
85952025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
85962025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 13 reason = Timeout
85972025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
85982025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 13 reason = Timeout
85992025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 25 reason = Timeout
86002025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 13 reason = Timeout
86012025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
86022025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
86032025-03-25T22:40:33.982ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 15 reason = Timeout
86042025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
86052025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
86062025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
86072025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 25 reason = Timeout
86082025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 25 reason = Timeout
86092025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 13 reason = Timeout
86102025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
86112025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86122025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
86132025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 25 reason = Timeout
86142025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
86152025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 13 reason = Timeout
86162025-03-25T22:40:33.983ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
86172025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
86182025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 25 reason = Timeout
86192025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
86202025-03-25T22:40:33.983ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("09fed906-08a5-4e51-9e85-fba7211b9594.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4ba21a7b-c053-4a77-807f-34395131fce2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1547, tv_nsec: 363509301 } })
86212025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 13 reason = Timeout
86222025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
86232025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
86242025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 6 reason = Timeout
86252025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
86262025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 25 reason = Timeout
86272025-03-25T22:40:33.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
86282025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 25 reason = Timeout
86292025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
86302025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
86312025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
86322025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 73 reason = Timeout
86332025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
86342025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 27 reason = Timeout
86352025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86362025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 25 reason = Timeout
86372025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
86382025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
86392025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 25 reason = Timeout
86402025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
86412025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 14 reason = Timeout
86422025-03-25T22:40:33.983ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b
86432025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 12.67\u{b5}s iteration = 14 status = {"enabled":false,"target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b"}
86442025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 13 reason = Timeout
86452025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 25 reason = Timeout
86462025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 25 reason = Timeout
86472025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
86482025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
86492025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
86502025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 73 reason = Timeout
86512025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 25 reason = Timeout
86522025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
86532025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
86542025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
86552025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 25 reason = Timeout
86562025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 25 reason = Timeout
86572025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
86582025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
86592025-03-25T22:40:33.983ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8f1103be-ae21-497e-8fb0-5373f3346d6f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1547, tv_nsec: 363798307 } })
86602025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
86612025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86622025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
86632025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
86642025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
86652025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
86662025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
86672025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
86682025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
86692025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.194032ms iteration = 13 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
86702025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
86712025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
86722025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
86732025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
86742025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
86752025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
86762025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
86772025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
86782025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
86792025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
86802025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
86812025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
86822025-03-25T22:40:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
86832025-03-25T22:40:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.145981ms iteration = 13 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
86842025-03-25T22:40:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
86852025-03-25T22:40:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
86862025-03-25T22:40:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
86872025-03-25T22:40:33.984ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
86882025-03-25T22:40:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
86892025-03-25T22:40:33.984ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
86902025-03-25T22:40:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
86912025-03-25T22:40:33.984ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
86922025-03-25T22:40:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
86932025-03-25T22:40:33.985ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
86942025-03-25T22:40:33.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
86952025-03-25T22:40:33.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1547, tv_nsec: 365932165 } })
86962025-03-25T22:40:33.985ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
86972025-03-25T22:40:33.985ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:30:33.982631398 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
86982025-03-25T22:40:33.985ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.28365ms iteration = 13 status = {"expiration":"2025-03-25T22:30:33.982631398Z","pruned":[]}
86992025-03-25T22:40:33.986ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.427662ms iteration = 25 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
87002025-03-25T22:40:33.986ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
87012025-03-25T22:40:33.986ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87022025-03-25T22:40:33.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
87032025-03-25T22:40:33.988ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1547, tv_nsec: 368361870 } })
87042025-03-25T22:40:33.988ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87052025-03-25T22:40:33.988ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
87062025-03-25T22:40:33.988ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87072025-03-25T22:40:33.990ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
87082025-03-25T22:40:33.990ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
87092025-03-25T22:40:33.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
87102025-03-25T22:40:33.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
87112025-03-25T22:40:33.991ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87122025-03-25T22:40:33.992ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
87132025-03-25T22:40:33.992ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
87142025-03-25T22:40:33.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87152025-03-25T22:40:33.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
87162025-03-25T22:40:33.993ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87172025-03-25T22:40:33.994ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87182025-03-25T22:40:33.994ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87192025-03-25T22:40:33.994ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87202025-03-25T22:40:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
87212025-03-25T22:40:33.996ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
87222025-03-25T22:40:33.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
87232025-03-25T22:40:33.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
87242025-03-25T22:40:33.997ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
87252025-03-25T22:40:33.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
87262025-03-25T22:40:33.998ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
87272025-03-25T22:40:33.998ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
87282025-03-25T22:40:33.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
87292025-03-25T22:40:33.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
87302025-03-25T22:40:33.998ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
87312025-03-25T22:40:33.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
87322025-03-25T22:40:34.000ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 16.905638ms iteration = 25 status = {"errors":[],"finish_invoked_ok":[]}
87332025-03-25T22:40:34.001ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87342025-03-25T22:40:34.004ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
87352025-03-25T22:40:34.004ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87362025-03-25T22:40:34.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87372025-03-25T22:40:34.007ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 24.320393ms iteration = 13 status = {"errors":[],"found_port_ok":[]}
87382025-03-25T22:40:34.008ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87392025-03-25T22:40:34.011ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
87402025-03-25T22:40:34.011ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
87412025-03-25T22:40:34.011ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87422025-03-25T22:40:34.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
87432025-03-25T22:40:34.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
87442025-03-25T22:40:34.014ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87452025-03-25T22:40:34.017ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
87462025-03-25T22:40:34.017ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
87472025-03-25T22:40:34.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
87482025-03-25T22:40:34.019ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87492025-03-25T22:40:34.023ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
87502025-03-25T22:40:34.023ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
87512025-03-25T22:40:34.023ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87522025-03-25T22:40:34.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
87532025-03-25T22:40:34.023ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87542025-03-25T22:40:34.026ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
87552025-03-25T22:40:34.026ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
87562025-03-25T22:40:34.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
87572025-03-25T22:40:34.026ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87582025-03-25T22:40:34.026ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
87592025-03-25T22:40:34.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
87602025-03-25T22:40:34.027ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87612025-03-25T22:40:34.027ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
87622025-03-25T22:40:34.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87632025-03-25T22:40:34.028ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
87642025-03-25T22:40:34.028ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
87652025-03-25T22:40:34.028ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
87662025-03-25T22:40:34.028ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
87672025-03-25T22:40:34.028ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
87682025-03-25T22:40:34.028ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
87692025-03-25T22:40:34.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
87702025-03-25T22:40:34.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
87712025-03-25T22:40:34.032ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1547, tv_nsec: 412808080 } })
87722025-03-25T22:40:34.034ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
87732025-03-25T22:40:34.034ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
87742025-03-25T22:40:34.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
87752025-03-25T22:40:34.035ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87762025-03-25T22:40:34.037ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
87772025-03-25T22:40:34.037ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
87782025-03-25T22:40:34.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
87792025-03-25T22:40:34.058ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 74.900936ms iteration = 25 status = {"errors":[],"garbage_collect_requested":[]}
87802025-03-25T22:40:34.058ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87812025-03-25T22:40:34.061ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87822025-03-25T22:40:34.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87832025-03-25T22:40:34.063ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
87842025-03-25T22:40:34.063ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
87852025-03-25T22:40:34.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
87862025-03-25T22:40:34.064ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
87872025-03-25T22:40:34.064ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87882025-03-25T22:40:34.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
87892025-03-25T22:40:34.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
87902025-03-25T22:40:34.067ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87912025-03-25T22:40:34.067ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87922025-03-25T22:40:34.071ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
87932025-03-25T22:40:34.071ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
87942025-03-25T22:40:34.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
87952025-03-25T22:40:34.072ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
87962025-03-25T22:40:34.072ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
87972025-03-25T22:40:34.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
87982025-03-25T22:40:34.074ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87992025-03-25T22:40:34.078ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
88002025-03-25T22:40:34.078ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
88012025-03-25T22:40:34.078ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 95.439111ms iteration = 13 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
88022025-03-25T22:40:34.078ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
88032025-03-25T22:40:34.080ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88042025-03-25T22:40:34.080ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
88052025-03-25T22:40:34.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
88062025-03-25T22:40:34.082ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
88072025-03-25T22:40:34.082ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
88082025-03-25T22:40:34.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
88092025-03-25T22:40:34.083ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
88102025-03-25T22:40:34.083ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
88112025-03-25T22:40:34.084ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
88122025-03-25T22:40:34.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
88132025-03-25T22:40:34.084ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 101.764086ms iteration = 25 status = {"errors":[],"requests_created_ok":[]}
88142025-03-25T22:40:34.087ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
88152025-03-25T22:40:34.087ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
88162025-03-25T22:40:34.087ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
88172025-03-25T22:40:34.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
88182025-03-25T22:40:34.088ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88192025-03-25T22:40:34.089ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88202025-03-25T22:40:34.102ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
88212025-03-25T22:40:34.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
88222025-03-25T22:40:34.114ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
88232025-03-25T22:40:34.114ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
88242025-03-25T22:40:34.114ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 131.244764ms iteration = 13 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
88252025-03-25T22:40:34.121ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
88262025-03-25T22:40:34.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
88272025-03-25T22:40:34.136ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
88282025-03-25T22:40:34.136ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
88292025-03-25T22:40:34.136ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
88302025-03-25T22:40:34.136ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
88312025-03-25T22:40:34.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
88322025-03-25T22:40:34.149ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
88332025-03-25T22:40:34.149ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
88342025-03-25T22:40:34.150ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc1f7178-5380-4036-8d4f-c300d790fb41", "content-length": "26", "date": "Tue, 25 Mar 2025 22:40:33 GMT"} })
88352025-03-25T22:40:34.150ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
88362025-03-25T22:40:34.152ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
88372025-03-25T22:40:34.154ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
88382025-03-25T22:40:34.154ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
88392025-03-25T22:40:34.154ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb20eea6-bb76-4563-ad08-46c04c5fc4ff", "content-length": "26", "date": "Tue, 25 Mar 2025 22:40:33 GMT"} })
88402025-03-25T22:40:34.154ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
88412025-03-25T22:40:34.155ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
88422025-03-25T22:40:34.246ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = instance_watcher body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vmms/09c83ce2-dbe9-4cd4-adaf-37bffcd806bf/state
88432025-03-25T22:40:34.252ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
88442025-03-25T22:40:34.252ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
88452025-03-25T22:40:34.252ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "34a45146-6bc5-45b7-8e91-5606e90083dc", "content-length": "26", "date": "Tue, 25 Mar 2025 22:40:33 GMT"} })
88462025-03-25T22:40:34.252ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
88472025-03-25T22:40:34.254ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
88482025-03-25T22:40:34.306ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
88492025-03-25T22:40:34.321ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
88502025-03-25T22:40:34.321ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
88512025-03-25T22:40:34.321ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2ffb01e-ea4d-4847-a433-0680b263eec9", "content-length": "583", "date": "Tue, 25 Mar 2025 22:40:33 GMT"} })
88522025-03-25T22:40:34.321ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88532025-03-25T22:40:34.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88542025-03-25T22:40:34.333ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 350.765397ms iteration = 6 status = {"nsuccess":5}
88552025-03-25T22:40:34.336ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88562025-03-25T22:40:34.337ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88572025-03-25T22:40:34.390ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
88582025-03-25T22:40:34.391ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
88592025-03-25T22:40:34.391ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
88602025-03-25T22:40:34.391ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
88612025-03-25T22:40:34.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
88622025-03-25T22:40:34.396ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
88632025-03-25T22:40:34.396ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 413.329999ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
88642025-03-25T22:40:34.396ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 413.561573ms iteration = 73 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
88652025-03-25T22:40:34.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
88662025-03-25T22:40:34.397ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
88672025-03-25T22:40:34.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
88682025-03-25T22:40:34.399ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88692025-03-25T22:40:34.401ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88702025-03-25T22:40:34.402ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88712025-03-25T22:40:34.403ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
88722025-03-25T22:40:34.403ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
88732025-03-25T22:40:34.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
88742025-03-25T22:40:34.407ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
88752025-03-25T22:40:34.407ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
88762025-03-25T22:40:34.407ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
88772025-03-25T22:40:34.407ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88782025-03-25T22:40:34.409ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88792025-03-25T22:40:34.409ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
88802025-03-25T22:40:34.409ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88812025-03-25T22:40:34.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88822025-03-25T22:40:34.410ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88832025-03-25T22:40:34.410ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 427.614299ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
88842025-03-25T22:40:34.411ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
88852025-03-25T22:40:34.411ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
88862025-03-25T22:40:34.411ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
88872025-03-25T22:40:34.413ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
88882025-03-25T22:40:34.413ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88892025-03-25T22:40:34.413ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
88902025-03-25T22:40:34.413ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
88912025-03-25T22:40:34.413ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
88922025-03-25T22:40:34.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88932025-03-25T22:40:34.413ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
88942025-03-25T22:40:34.413ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
88952025-03-25T22:40:34.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
88962025-03-25T22:40:34.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
88972025-03-25T22:40:34.416ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 433.789102ms iteration = 25 status = 0
88982025-03-25T22:40:34.419ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
88992025-03-25T22:40:34.419ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "618eff07-1549-42d7-9495-45884bec95d2", "content-length": "1", "date": "Tue, 25 Mar 2025 22:40:33 GMT"} })
89002025-03-25T22:40:34.419ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89012025-03-25T22:40:34.421ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
89022025-03-25T22:40:34.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
89032025-03-25T22:40:34.424ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
89042025-03-25T22:40:34.424ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
89052025-03-25T22:40:34.424ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
89062025-03-25T22:40:34.425ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
89072025-03-25T22:40:34.425ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 442.626283ms iteration = 73 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
89082025-03-25T22:40:34.426ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 444.335203ms iteration = 25 status = null
89092025-03-25T22:40:34.437ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 454.375977ms iteration = 25 status = {}
89102025-03-25T22:40:34.445ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89112025-03-25T22:40:34.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
89122025-03-25T22:40:34.449ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
89132025-03-25T22:40:34.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
89142025-03-25T22:40:34.459ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
89152025-03-25T22:40:34.459ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 476.270136ms iteration = 25 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
89162025-03-25T22:40:34.459ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
89172025-03-25T22:40:34.459ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:29:06.674111 UTC dns_group = internal generation = 1
89182025-03-25T22:40:34.459ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 476.950279ms iteration = 13 status = {"generation":1}
89192025-03-25T22:40:34.464ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:28:31.652727Z, time_modified: 2025-03-25T22:28:31.652727Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
89202025-03-25T22:40:34.464ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 482.040301ms iteration = 13 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]}},"default_endpoint":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
89212025-03-25T22:40:34.464ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {8afb3eab-ff69-4f48-8513-92fd27ac01b5: {10.151.2.176: {a606dc76-3d9b-43e0-a4a5-ec794211425e}, 10.151.2.175: {a606dc76-3d9b-43e0-a4a5-ec794211425e}}} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
89222025-03-25T22:40:34.464ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
89232025-03-25T22:40:34.465ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "a3c0301a-3b52-4580-9137-328fefa080b6", "date": "Tue, 25 Mar 2025 22:40:34 GMT"} })
89242025-03-25T22:40:34.465ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89252025-03-25T22:40:34.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89262025-03-25T22:40:34.468ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89272025-03-25T22:40:34.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89282025-03-25T22:40:34.472ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89292025-03-25T22:40:34.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89302025-03-25T22:40:34.475ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
89312025-03-25T22:40:34.475ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
89322025-03-25T22:40:34.475ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 492.025704ms iteration = 25 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
89332025-03-25T22:40:34.476ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
89342025-03-25T22:40:34.476ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 493.036172ms iteration = 25 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
89352025-03-25T22:40:34.477ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89362025-03-25T22:40:34.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89372025-03-25T22:40:34.483ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89382025-03-25T22:40:34.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89392025-03-25T22:40:34.485ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
89402025-03-25T22:40:34.485ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 0 version = 2
89412025-03-25T22:40:34.485ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
89422025-03-25T22:40:34.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
89432025-03-25T22:40:34.496ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89442025-03-25T22:40:34.496ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89452025-03-25T22:40:34.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89462025-03-25T22:40:34.502ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
89472025-03-25T22:40:34.502ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
89482025-03-25T22:40:34.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
89492025-03-25T22:40:34.503ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
89502025-03-25T22:40:34.503ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
89512025-03-25T22:40:34.503ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
89522025-03-25T22:40:34.503ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "5dc0e7cd-fe4d-4850-84fd-f4ed8f3f8dd9", "date": "Tue, 25 Mar 2025 22:40:34 GMT"} })
89532025-03-25T22:40:34.505ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 521.953719ms iteration = 25 status = {}
89542025-03-25T22:40:34.505ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 1 version = 2
89552025-03-25T22:40:34.505ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
89562025-03-25T22:40:34.505ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:29:07.710706 UTC dns_group = external generation = 2
89572025-03-25T22:40:34.505ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 523.111481ms iteration = 13 status = {"generation":2}
89582025-03-25T22:40:34.510ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
89592025-03-25T22:40:34.515ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
89602025-03-25T22:40:34.515ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 532.278239ms iteration = 13 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
89612025-03-25T22:40:34.520ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 537.173258ms iteration = 27 status = {"physical_disks_added":0}
89622025-03-25T22:40:34.668ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
89632025-03-25T22:40:34.669ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "511743de-7c3b-41ae-aa37-56932fa3c029", "content-length": "810", "date": "Tue, 25 Mar 2025 22:40:34 GMT"} })
89642025-03-25T22:40:34.669ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {VirtualNetworkInterfaceHost { physical_host_ip: fd00:1122:3344:101::1, virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), vni: Vni(6019492) }} sled = sock
89652025-03-25T22:40:34.669ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
89662025-03-25T22:40:34.669ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
89672025-03-25T22:40:34.673ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
89682025-03-25T22:40:34.673ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 690.743178ms iteration = 25 status = {}
89692025-03-25T22:40:34.674ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "e1c9e911-0881-4cb8-9b3f-d51e2ec4c712", "date": "Tue, 25 Mar 2025 22:40:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
89702025-03-25T22:40:34.674ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
89712025-03-25T22:40:34.677ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "4d0d509e-aae9-400e-a9f9-0522ec098c55", "date": "Tue, 25 Mar 2025 22:40:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
89722025-03-25T22:40:34.677ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
89732025-03-25T22:40:34.679ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
89742025-03-25T22:40:34.679ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 696.251959ms iteration = 15 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
89752025-03-25T22:40:34.694ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89762025-03-25T22:40:34.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89772025-03-25T22:40:34.696ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
89782025-03-25T22:40:34.697ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89792025-03-25T22:40:34.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89802025-03-25T22:40:34.700ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
89812025-03-25T22:40:34.700ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "fff0c5db-aa44-4bf0-962f-cc67436d95cd", "date": "Tue, 25 Mar 2025 22:40:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89822025-03-25T22:40:34.700ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89832025-03-25T22:40:34.702ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "aca570b2-00d1-4349-9b54-cea46af810d0", "date": "Tue, 25 Mar 2025 22:40:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89842025-03-25T22:40:34.702ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89852025-03-25T22:40:34.704ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "3384297a-dd83-4a3e-af4c-dfbe63c25cd4", "date": "Tue, 25 Mar 2025 22:40:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89862025-03-25T22:40:34.704ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89872025-03-25T22:40:34.706ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
89882025-03-25T22:40:34.706ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 723.393243ms iteration = 15 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
89892025-03-25T22:40:34.707ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
89902025-03-25T22:40:34.707ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "c64f1ac1-a47f-4918-b3ba-e65b847f810f", "content-length": "134", "date": "Tue, 25 Mar 2025 22:40:34 GMT"} })
89912025-03-25T22:40:34.707ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
89922025-03-25T22:40:34.707ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
89932025-03-25T22:40:34.708ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
89942025-03-25T22:40:34.708ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
89952025-03-25T22:40:34.708ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89962025-03-25T22:40:34.708ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89972025-03-25T22:40:34.708ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89982025-03-25T22:40:34.708ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
89992025-03-25T22:40:34.708ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
90002025-03-25T22:40:34.708ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
90012025-03-25T22:40:34.708ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
90022025-03-25T22:40:34.708ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90032025-03-25T22:40:34.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90042025-03-25T22:40:34.710ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
90052025-03-25T22:40:34.710ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "d873c97e-c5dd-4370-90e7-9e3b135c9291", "content-length": "2", "date": "Tue, 25 Mar 2025 22:40:34 GMT"} })
90062025-03-25T22:40:34.710ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
90072025-03-25T22:40:34.710ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "e987cbaa-7555-4836-b55b-2aed9703c763", "content-length": "52", "date": "Tue, 25 Mar 2025 22:40:34 GMT"} })
90082025-03-25T22:40:34.710ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
90092025-03-25T22:40:34.710ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
90102025-03-25T22:40:34.821ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "0ec65828-f536-4fc0-bc28-a8c898b8ab0e", "date": "Tue, 25 Mar 2025 22:40:34 GMT"} })
90112025-03-25T22:40:34.822ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
90122025-03-25T22:40:34.822ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
90132025-03-25T22:40:34.826ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1636cc5-ddb6-4b40-9268-368fd4d51ddb", "content-length": "682", "date": "Tue, 25 Mar 2025 22:40:34 GMT"} })
90142025-03-25T22:40:34.826ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90152025-03-25T22:40:34.826ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90162025-03-25T22:40:34.830ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90172025-03-25T22:40:34.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90182025-03-25T22:40:34.834ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90192025-03-25T22:40:34.834ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90202025-03-25T22:40:34.836ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90212025-03-25T22:40:34.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90222025-03-25T22:40:34.844ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 861.173387ms iteration = 25 status = {}
90232025-03-25T22:40:36.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 302 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 532629bf-d20e-4b25-a5fc-be6b9c3efa40 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
90242025-03-25T22:40:36.129ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = instance_watcher result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vmms/09c83ce2-dbe9-4cd4-adaf-37bffcd806bf/state", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0044f89-da5a-44fa-a4f6-227010dae161", "content-length": "132", "date": "Tue, 25 Mar 2025 22:40:36 GMT"} })
90252025-03-25T22:40:36.136ZDEBGnexus (BackgroundTasks): updating instance state background_task = instance_watcher instance_id = 5db019c6-f069-4c05-ac23-244eb3969e9b sled_id = f7ac1017-36e7-40ec-9172-e4e1545e73c8 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2025-03-25T22:40:36.039128364Z }, migration_in: None, migration_out: None } vmm_id = 09c83ce2-dbe9-4cd4-adaf-37bffcd806bf
90262025-03-25T22:40:36.136ZINFOnexus (BackgroundTasks): received new VMM runtime state from sled agent background_task = instance_watcher file = nexus/src/app/instance.rs:2120 instance_id = 5db019c6-f069-4c05-ac23-244eb3969e9b migration_state = Migrations { migration_in: None, migration_out: None } propolis_id = 09c83ce2-dbe9-4cd4-adaf-37bffcd806bf sled_id = f7ac1017-36e7-40ec-9172-e4e1545e73c8 vmm_id = 09c83ce2-dbe9-4cd4-adaf-37bffcd806bf vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2025-03-25T22:40:36.039128364Z }
90272025-03-25T22:40:36.136ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 5db019c6-f069-4c05-ac23-244eb3969e9b roles = RoleSet { roles: {} } sled_id = f7ac1017-36e7-40ec-9172-e4e1545e73c8 vmm_id = 09c83ce2-dbe9-4cd4-adaf-37bffcd806bf
90282025-03-25T22:40:36.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 5db019c6-f069-4c05-ac23-244eb3969e9b resource = Database result = Ok(()) sled_id = f7ac1017-36e7-40ec-9172-e4e1545e73c8 vmm_id = 09c83ce2-dbe9-4cd4-adaf-37bffcd806bf
90292025-03-25T22:40:36.154ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 1 total_failed = 0 total_incomplete = 0 total_instances = 1 update_sagas_queued = 0
90302025-03-25T22:40:36.156ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 2.173405749s iteration = 25 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{"destroyed":1},"pruned_instances":0,"total_instances":1,"update_sagas_queued":0}
90312025-03-25T22:40:43.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 74 reason = Timeout
90322025-03-25T22:40:43.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 74 reason = Timeout
90332025-03-25T22:40:43.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90342025-03-25T22:40:43.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90352025-03-25T22:40:43.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90362025-03-25T22:40:43.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90372025-03-25T22:40:43.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90382025-03-25T22:40:43.989ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
90392025-03-25T22:40:43.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90402025-03-25T22:40:43.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90412025-03-25T22:40:43.991ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90422025-03-25T22:40:43.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90432025-03-25T22:40:44.019ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90442025-03-25T22:40:44.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90452025-03-25T22:40:44.023ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90462025-03-25T22:40:44.026ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
90472025-03-25T22:40:44.026ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90482025-03-25T22:40:44.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90492025-03-25T22:40:44.035ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
90502025-03-25T22:40:44.035ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 51.836875ms iteration = 74 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
90512025-03-25T22:40:44.048ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.606518ms iteration = 74 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
90522025-03-25T22:40:46.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 313 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = b6ef24b8-069e-4006-9009-26dd495f01a2 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
90532025-03-25T22:40:53.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 75 reason = Timeout
90542025-03-25T22:40:53.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 75 reason = Timeout
90552025-03-25T22:40:53.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90562025-03-25T22:40:53.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90572025-03-25T22:40:53.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90582025-03-25T22:40:53.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90592025-03-25T22:40:54.014ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90602025-03-25T22:40:54.017ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
90612025-03-25T22:40:54.017ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90622025-03-25T22:40:54.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90632025-03-25T22:40:54.019ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90642025-03-25T22:40:54.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90652025-03-25T22:40:54.023ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90662025-03-25T22:40:54.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90672025-03-25T22:40:54.025ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 41.832013ms iteration = 75 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
90682025-03-25T22:40:54.048ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90692025-03-25T22:40:54.050ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
90702025-03-25T22:40:54.050ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90712025-03-25T22:40:54.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90722025-03-25T22:40:54.060ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
90732025-03-25T22:40:54.060ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 77.071225ms iteration = 75 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
90742025-03-25T22:40:56.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 312 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = a67325d5-d8ac-4a7d-93db-649b28e93813 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
90752025-03-25T22:41:03.982ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 26 reason = Timeout
90762025-03-25T22:41:03.982ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 26 reason = Timeout
90772025-03-25T22:41:03.982ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 26 reason = Timeout
90782025-03-25T22:41:03.982ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
90792025-03-25T22:41:03.982ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
90802025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
90812025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
90822025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 26 reason = Timeout
90832025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 26 reason = Timeout
90842025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 26 reason = Timeout
90852025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 26 reason = Timeout
90862025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
90872025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
90882025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
90892025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
90902025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 26 reason = Timeout
90912025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 26 reason = Timeout
90922025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
90932025-03-25T22:41:03.983ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
90942025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 26 reason = Timeout
90952025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90962025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 76 reason = Timeout
90972025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 26 reason = Timeout
90982025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90992025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 28 reason = Timeout
91002025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
91012025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 26 reason = Timeout
91022025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
91032025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
91042025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 26 reason = Timeout
91052025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
91062025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 26 reason = Timeout
91072025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
91082025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
91092025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 76 reason = Timeout
91102025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
91112025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 26 reason = Timeout
91122025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
91132025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 26 reason = Timeout
91142025-03-25T22:41:03.984ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
91152025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91162025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
91172025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
91182025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
91192025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
91202025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
91212025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91222025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
91232025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
91242025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
91252025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
91262025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
91272025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
91282025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91292025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
91302025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
91312025-03-25T22:41:03.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
91322025-03-25T22:41:03.985ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1577, tv_nsec: 365937826 } })
91332025-03-25T22:41:03.985ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
91342025-03-25T22:41:03.986ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 2.176929ms iteration = 26 status = {"errors":[],"finish_invoked_ok":[]}
91352025-03-25T22:41:03.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1577, tv_nsec: 367032825 } })
91362025-03-25T22:41:03.986ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
91372025-03-25T22:41:03.987ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.539312ms iteration = 26 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
91382025-03-25T22:41:03.987ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91392025-03-25T22:41:03.988ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91402025-03-25T22:41:03.989ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91412025-03-25T22:41:03.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
91422025-03-25T22:41:03.991ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
91432025-03-25T22:41:03.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
91442025-03-25T22:41:03.992ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
91452025-03-25T22:41:03.992ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
91462025-03-25T22:41:03.992ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
91472025-03-25T22:41:03.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91482025-03-25T22:41:03.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
91492025-03-25T22:41:03.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91502025-03-25T22:41:03.995ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
91512025-03-25T22:41:03.995ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
91522025-03-25T22:41:03.995ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
91532025-03-25T22:41:03.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
91542025-03-25T22:41:04.006ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 22.807206ms iteration = 26 status = {"errors":[],"requests_created_ok":[]}
91552025-03-25T22:41:04.007ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91562025-03-25T22:41:04.007ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
91572025-03-25T22:41:04.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
91582025-03-25T22:41:04.010ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
91592025-03-25T22:41:04.010ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
91602025-03-25T22:41:04.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
91612025-03-25T22:41:04.024ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91622025-03-25T22:41:04.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91632025-03-25T22:41:04.029ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
91642025-03-25T22:41:04.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
91652025-03-25T22:41:04.032ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1577, tv_nsec: 413291279 } })
91662025-03-25T22:41:04.033ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
91672025-03-25T22:41:04.048ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 64.449565ms iteration = 26 status = {"errors":[],"garbage_collect_requested":[]}
91682025-03-25T22:41:04.051ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91692025-03-25T22:41:04.052ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91702025-03-25T22:41:04.054ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
91712025-03-25T22:41:04.054ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
91722025-03-25T22:41:04.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
91732025-03-25T22:41:04.056ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
91742025-03-25T22:41:04.056ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
91752025-03-25T22:41:04.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
91762025-03-25T22:41:04.057ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
91772025-03-25T22:41:04.057ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
91782025-03-25T22:41:04.059ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 75.64662ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
91792025-03-25T22:41:04.061ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91802025-03-25T22:41:04.064ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
91812025-03-25T22:41:04.064ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91822025-03-25T22:41:04.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91832025-03-25T22:41:04.064ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 1 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
91842025-03-25T22:41:04.064ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 80.958476ms iteration = 26 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":1,"total_instances":0,"update_sagas_queued":0}
91852025-03-25T22:41:04.064ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
91862025-03-25T22:41:04.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
91872025-03-25T22:41:04.066ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
91882025-03-25T22:41:04.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
91892025-03-25T22:41:04.066ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91902025-03-25T22:41:04.067ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
91912025-03-25T22:41:04.067ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
91922025-03-25T22:41:04.068ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
91932025-03-25T22:41:04.068ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91942025-03-25T22:41:04.068ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
91952025-03-25T22:41:04.068ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
91962025-03-25T22:41:04.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91972025-03-25T22:41:04.068ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "248e82f3-37a0-4758-8740-f58f375fb8ad", "content-length": "26", "date": "Tue, 25 Mar 2025 22:41:04 GMT"} })
91982025-03-25T22:41:04.068ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
91992025-03-25T22:41:04.068ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 84.942269ms iteration = 76 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
92002025-03-25T22:41:04.070ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
92012025-03-25T22:41:04.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
92022025-03-25T22:41:04.070ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92032025-03-25T22:41:04.071ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
92042025-03-25T22:41:04.072ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
92052025-03-25T22:41:04.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
92062025-03-25T22:41:04.074ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92072025-03-25T22:41:04.074ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
92082025-03-25T22:41:04.074ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92092025-03-25T22:41:04.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92102025-03-25T22:41:04.076ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
92112025-03-25T22:41:04.076ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
92122025-03-25T22:41:04.076ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
92132025-03-25T22:41:04.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
92142025-03-25T22:41:04.078ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
92152025-03-25T22:41:04.078ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
92162025-03-25T22:41:04.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
92172025-03-25T22:41:04.080ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92182025-03-25T22:41:04.080ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 96.792595ms iteration = 26 status = 0
92192025-03-25T22:41:04.081ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
92202025-03-25T22:41:04.081ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
92212025-03-25T22:41:04.081ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5b0d652-c185-459d-86c6-ac157ab44e28", "content-length": "26", "date": "Tue, 25 Mar 2025 22:41:04 GMT"} })
92222025-03-25T22:41:04.081ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
92232025-03-25T22:41:04.082ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
92242025-03-25T22:41:04.082ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 98.868593ms iteration = 76 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
92252025-03-25T22:41:04.083ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
92262025-03-25T22:41:04.083ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
92272025-03-25T22:41:04.083ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
92282025-03-25T22:41:04.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
92292025-03-25T22:41:04.097ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 113.240226ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
92302025-03-25T22:41:04.103ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
92312025-03-25T22:41:04.104ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
92322025-03-25T22:41:04.105ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92332025-03-25T22:41:04.108ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
92342025-03-25T22:41:04.108ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
92352025-03-25T22:41:04.109ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
92362025-03-25T22:41:04.110ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
92372025-03-25T22:41:04.110ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
92382025-03-25T22:41:04.110ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 126.296624ms iteration = 26 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
92392025-03-25T22:41:04.110ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
92402025-03-25T22:41:04.110ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
92412025-03-25T22:41:04.139ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 155.970764ms iteration = 28 status = {"physical_disks_added":0}
92422025-03-25T22:41:04.141ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
92432025-03-25T22:41:04.141ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
92442025-03-25T22:41:04.141ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "80a8a3aa-7816-4959-b5a9-12b5719a7e6f", "content-length": "26", "date": "Tue, 25 Mar 2025 22:41:04 GMT"} })
92452025-03-25T22:41:04.141ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
92462025-03-25T22:41:04.141ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
92472025-03-25T22:41:04.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
92482025-03-25T22:41:04.143ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
92492025-03-25T22:41:04.155ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 171.674371ms iteration = 26 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
92502025-03-25T22:41:04.155ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
92512025-03-25T22:41:04.155ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 171.930656ms iteration = 26 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
92522025-03-25T22:41:04.158ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 175.957149ms iteration = 26 status = {}
92532025-03-25T22:41:04.172ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
92542025-03-25T22:41:04.172ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd44ec6d-d2fb-477f-9cb6-0e09869e6c01", "content-length": "2", "date": "Tue, 25 Mar 2025 22:41:03 GMT"} })
92552025-03-25T22:41:04.174ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
92562025-03-25T22:41:04.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
92572025-03-25T22:41:04.176ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 193.578411ms iteration = 26 status = null
92582025-03-25T22:41:04.243ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
92592025-03-25T22:41:04.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
92602025-03-25T22:41:04.245ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92612025-03-25T22:41:04.247ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
92622025-03-25T22:41:04.247ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
92632025-03-25T22:41:04.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
92642025-03-25T22:41:04.249ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92652025-03-25T22:41:04.252ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
92662025-03-25T22:41:04.252ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
92672025-03-25T22:41:04.252ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
92682025-03-25T22:41:04.265ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
92692025-03-25T22:41:04.265ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
92702025-03-25T22:41:04.266ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d5bfa48-26cf-46f0-954d-9adb31d73e05", "content-length": "372", "date": "Tue, 25 Mar 2025 22:41:04 GMT"} })
92712025-03-25T22:41:04.266ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
92722025-03-25T22:41:04.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
92732025-03-25T22:41:04.312ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
92742025-03-25T22:41:04.312ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
92752025-03-25T22:41:04.325ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92762025-03-25T22:41:04.326ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92772025-03-25T22:41:04.328ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
92782025-03-25T22:41:04.328ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92792025-03-25T22:41:04.329ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92802025-03-25T22:41:04.354ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
92812025-03-25T22:41:04.355ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "3002ca61-b19a-4884-b64b-ce11dc427feb", "content-length": "134", "date": "Tue, 25 Mar 2025 22:41:04 GMT"} })
92822025-03-25T22:41:04.355ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
92832025-03-25T22:41:04.355ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
92842025-03-25T22:41:04.355ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
92852025-03-25T22:41:04.355ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
92862025-03-25T22:41:04.355ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
92872025-03-25T22:41:04.355ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
92882025-03-25T22:41:04.355ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
92892025-03-25T22:41:04.355ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
92902025-03-25T22:41:04.355ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
92912025-03-25T22:41:04.355ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
92922025-03-25T22:41:04.356ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
92932025-03-25T22:41:04.356ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92942025-03-25T22:41:04.356ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92952025-03-25T22:41:04.371ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
92962025-03-25T22:41:04.374ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
92972025-03-25T22:41:04.374ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "108c8f06-5d95-4444-9941-60aca2ce51ee", "content-length": "2", "date": "Tue, 25 Mar 2025 22:41:04 GMT"} })
92982025-03-25T22:41:04.374ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
92992025-03-25T22:41:04.374ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "e508acf5-6a39-4d94-98b1-ce7c777b85bc", "content-length": "52", "date": "Tue, 25 Mar 2025 22:41:04 GMT"} })
93002025-03-25T22:41:04.374ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
93012025-03-25T22:41:04.374ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
93022025-03-25T22:41:04.436ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "601a5e50-6a2e-4fe9-9705-3bdf33c4623d", "content-length": "691", "date": "Tue, 25 Mar 2025 22:41:04 GMT"} })
93032025-03-25T22:41:04.437ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
93042025-03-25T22:41:04.437ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
93052025-03-25T22:41:04.437ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
93062025-03-25T22:41:04.437ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "f0880db1-d2d9-4afd-9304-afae3a5f6e3b", "date": "Tue, 25 Mar 2025 22:41:04 GMT"} })
93072025-03-25T22:41:04.437ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
93082025-03-25T22:41:04.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
93092025-03-25T22:41:04.439ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 455.466757ms iteration = 26 status = {}
93102025-03-25T22:41:04.440ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
93112025-03-25T22:41:04.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
93122025-03-25T22:41:04.443ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
93132025-03-25T22:41:04.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
93142025-03-25T22:41:04.451ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
93152025-03-25T22:41:04.451ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
93162025-03-25T22:41:04.452ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "830de688-a96d-42d6-8ab4-13cc1939369a", "date": "Tue, 25 Mar 2025 22:41:04 GMT"} })
93172025-03-25T22:41:04.454ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 470.616963ms iteration = 26 status = {}
93182025-03-25T22:41:04.542ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "df427470-f49f-4b69-acff-59462f3d8848", "date": "Tue, 25 Mar 2025 22:41:04 GMT"} })
93192025-03-25T22:41:04.542ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
93202025-03-25T22:41:04.542ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
93212025-03-25T22:41:04.543ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "a453a87a-6513-4511-82d4-3327f61e3778", "content-length": "682", "date": "Tue, 25 Mar 2025 22:41:04 GMT"} })
93222025-03-25T22:41:04.543ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
93232025-03-25T22:41:04.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
93242025-03-25T22:41:04.547ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
93252025-03-25T22:41:04.547ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
93262025-03-25T22:41:04.550ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
93272025-03-25T22:41:04.550ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
93282025-03-25T22:41:04.552ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
93292025-03-25T22:41:04.552ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
93302025-03-25T22:41:04.559ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 575.409575ms iteration = 26 status = {}
93312025-03-25T22:41:06.085ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 257 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = e2ab257c-e96a-4d8b-89e9-ddccb125ecdc response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
93322025-03-25T22:41:13.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 77 reason = Timeout
93332025-03-25T22:41:13.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 77 reason = Timeout
93342025-03-25T22:41:13.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
93352025-03-25T22:41:13.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93362025-03-25T22:41:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
93372025-03-25T22:41:13.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93382025-03-25T22:41:13.986ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93392025-03-25T22:41:13.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
93402025-03-25T22:41:13.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93412025-03-25T22:41:13.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93422025-03-25T22:41:13.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93432025-03-25T22:41:13.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93442025-03-25T22:41:14.015ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
93452025-03-25T22:41:14.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
93462025-03-25T22:41:14.022ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93472025-03-25T22:41:14.025ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
93482025-03-25T22:41:14.025ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93492025-03-25T22:41:14.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93502025-03-25T22:41:14.033ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
93512025-03-25T22:41:14.033ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 50.365039ms iteration = 77 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
93522025-03-25T22:41:14.045ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.279206ms iteration = 77 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
93532025-03-25T22:41:16.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 265 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = c670a853-bb93-4750-a4d0-d952350295e9 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
93542025-03-25T22:41:23.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 78 reason = Timeout
93552025-03-25T22:41:23.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
93562025-03-25T22:41:23.984ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 78 reason = Timeout
93572025-03-25T22:41:23.984ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93582025-03-25T22:41:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
93592025-03-25T22:41:23.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93602025-03-25T22:41:24.015ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
93612025-03-25T22:41:24.015ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93622025-03-25T22:41:24.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
93632025-03-25T22:41:24.018ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
93642025-03-25T22:41:24.018ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93652025-03-25T22:41:24.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93662025-03-25T22:41:24.020ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93672025-03-25T22:41:24.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93682025-03-25T22:41:24.022ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93692025-03-25T22:41:24.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
93702025-03-25T22:41:24.024ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93712025-03-25T22:41:24.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93722025-03-25T22:41:24.033ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
93732025-03-25T22:41:24.033ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 49.555104ms iteration = 78 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
93742025-03-25T22:41:24.045ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 61.810808ms iteration = 78 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
93752025-03-25T22:41:26.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 428 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = f7ea3af9-f07c-419e-9ac1-e53cb8dbbe90 response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
93762025-03-25T22:41:31.078ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:62130
93772025-03-25T22:41:31.078ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62130 req_id = 60131f7d-cebc-4d6e-9ff6-7fb954439937 roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
93782025-03-25T22:41:31.079ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62130 req_id = 60131f7d-cebc-4d6e-9ff6-7fb954439937 resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
93792025-03-25T22:41:31.081ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2734 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62130 req_id = 60131f7d-cebc-4d6e-9ff6-7fb954439937 response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?sort_by=id_ascending
93802025-03-25T22:41:31.081ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62130 req_id = dd3cdc03-655a-440f-8c7d-02fdae3c68af roles = RoleSet { roles: {} } uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
93812025-03-25T22:41:31.082ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62130 req_id = dd3cdc03-655a-440f-8c7d-02fdae3c68af resource = Database result = Ok(()) uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
93822025-03-25T22:41:31.084ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2910 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62130 req_id = dd3cdc03-655a-440f-8c7d-02fdae3c68af response_code = 200 uri = /metrics/collectors/26728504-e36d-4baa-856d-cd0ce7a172a2/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2FjMTAxNy0zNmU3LTQwZWMtOTE3Mi1lNGUxNTQ1ZTczYzgifX0%3D
93832025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 14 reason = Timeout
93842025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 27 reason = Timeout
93852025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
93862025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 16 reason = Timeout
93872025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 14 reason = Timeout
93882025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 16 reason = Timeout
93892025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
93902025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 14 reason = Timeout
93912025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
93922025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
93932025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 14 reason = Timeout
93942025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 14 reason = Timeout
93952025-03-25T22:41:33.983ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-25 22:31:33.983299939 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
93962025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
93972025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 27 reason = Timeout
93982025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
93992025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
94002025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 27 reason = Timeout
94012025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 14 reason = Timeout
94022025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
94032025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 27 reason = Timeout
94042025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
94052025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 79 reason = Timeout
94062025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
94072025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 27 reason = Timeout
94082025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 27 reason = Timeout
94092025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
94102025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 27 reason = Timeout
94112025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
94122025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
94132025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 27 reason = Timeout
94142025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
94152025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 14 reason = Timeout
94162025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
94172025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 27 reason = Timeout
94182025-03-25T22:41:33.983ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
94192025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 27 reason = Timeout
94202025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
94212025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 79 reason = Timeout
94222025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
94232025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
94242025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 27 reason = Timeout
94252025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 27 reason = Timeout
94262025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
94272025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
94282025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 27 reason = Timeout
94292025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
94302025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 14 reason = Timeout
94312025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
94322025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 14 reason = Timeout
94332025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
94342025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 27 reason = Timeout
94352025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
94362025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
94372025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 29 reason = Timeout
94382025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
94392025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
94402025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 27 reason = Timeout
94412025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
94422025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 15 reason = Timeout
94432025-03-25T22:41:33.983ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b
94442025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.19\u{b5}s iteration = 15 status = {"enabled":false,"target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b"}
94452025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 14 reason = Timeout
94462025-03-25T22:41:33.983ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
94472025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 27 reason = Timeout
94482025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
94492025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
94502025-03-25T22:41:33.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
94512025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
94522025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
94532025-03-25T22:41:33.984ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("09fed906-08a5-4e51-9e85-fba7211b9594.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4ba21a7b-c053-4a77-807f-34395131fce2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cd12083a-1748-4471-a9f6-97e6c7aebe80.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1607, tv_nsec: 364283035 } })
94542025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
94552025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
94562025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
94572025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
94582025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
94592025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
94602025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
94612025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
94622025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
94632025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
94642025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
94652025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
94662025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
94672025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
94682025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
94692025-03-25T22:41:33.984ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8f1103be-ae21-497e-8fb0-5373f3346d6f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8c65163-9425-400d-ac5c-65dfc41d0bf4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1607, tv_nsec: 364515819 } })
94702025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
94712025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
94722025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
94732025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
94742025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
94752025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
94762025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
94772025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
94782025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.522808ms iteration = 14 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
94792025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
94802025-03-25T22:41:33.984ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.478637ms iteration = 14 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
94812025-03-25T22:41:33.986ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
94822025-03-25T22:41:33.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
94832025-03-25T22:41:33.986ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1607, tv_nsec: 366944664 } })
94842025-03-25T22:41:33.987ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1077
94852025-03-25T22:41:33.988ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94862025-03-25T22:41:33.989ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1607, tv_nsec: 369572492 } })
94872025-03-25T22:41:33.989ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 6.160252ms iteration = 27 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
94882025-03-25T22:41:33.989ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1077
94892025-03-25T22:41:33.991ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
94902025-03-25T22:41:33.991ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
94912025-03-25T22:41:33.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
94922025-03-25T22:41:33.993ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94932025-03-25T22:41:33.993ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-25 22:31:33.983299939 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
94942025-03-25T22:41:33.993ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 10.577533ms iteration = 14 status = {"expiration":"2025-03-25T22:31:33.983299939Z","pruned":[]}
94952025-03-25T22:41:33.994ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
94962025-03-25T22:41:33.994ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 10.869519ms iteration = 27 status = {"errors":[],"garbage_collect_requested":[]}
94972025-03-25T22:41:33.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
94982025-03-25T22:41:33.994ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94992025-03-25T22:41:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
95002025-03-25T22:41:33.996ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
95012025-03-25T22:41:33.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
95022025-03-25T22:41:33.998ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
95032025-03-25T22:41:33.998ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95042025-03-25T22:41:33.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95052025-03-25T22:41:34.003ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
95062025-03-25T22:41:34.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
95072025-03-25T22:41:34.004ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 20.902441ms iteration = 14 status = {"errors":[],"found_port_ok":[]}
95082025-03-25T22:41:34.008ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
95092025-03-25T22:41:34.008ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
95102025-03-25T22:41:34.008ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
95112025-03-25T22:41:34.013ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
95122025-03-25T22:41:34.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
95132025-03-25T22:41:34.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
95142025-03-25T22:41:34.035ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
95152025-03-25T22:41:34.035ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 51.746874ms iteration = 27 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
95162025-03-25T22:41:34.060ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95172025-03-25T22:41:34.062ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
95182025-03-25T22:41:34.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
95192025-03-25T22:41:34.064ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
95202025-03-25T22:41:34.064ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
95212025-03-25T22:41:34.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
95222025-03-25T22:41:34.065ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
95232025-03-25T22:41:34.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
95242025-03-25T22:41:34.067ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95252025-03-25T22:41:34.070ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
95262025-03-25T22:41:34.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
95272025-03-25T22:41:34.071ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
95282025-03-25T22:41:34.071ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95292025-03-25T22:41:34.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95302025-03-25T22:41:34.072ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95312025-03-25T22:41:34.072ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95322025-03-25T22:41:34.074ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95332025-03-25T22:41:34.077ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 94.29965ms iteration = 27 status = {"errors":[],"requests_created_ok":[]}
95342025-03-25T22:41:34.077ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
95352025-03-25T22:41:34.077ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
95362025-03-25T22:41:34.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
95372025-03-25T22:41:34.078ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
95382025-03-25T22:41:34.078ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
95392025-03-25T22:41:34.079ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
95402025-03-25T22:41:34.079ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
95412025-03-25T22:41:34.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
95422025-03-25T22:41:34.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
95432025-03-25T22:41:34.084ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 100.806938ms iteration = 27 status = {"errors":[],"finish_invoked_ok":[]}
95442025-03-25T22:41:34.084ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
95452025-03-25T22:41:34.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
95462025-03-25T22:41:34.089ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
95472025-03-25T22:41:34.089ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
95482025-03-25T22:41:34.092ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
95492025-03-25T22:41:34.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
95502025-03-25T22:41:34.093ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f7ac1017-36e7-40ec-9172-e4e1545e73c8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1607, tv_nsec: 473529128 } })
95512025-03-25T22:41:34.093ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95522025-03-25T22:41:34.093ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1077 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
95532025-03-25T22:41:34.095ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
95542025-03-25T22:41:34.095ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
95552025-03-25T22:41:34.095ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 112.065364ms iteration = 14 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
95562025-03-25T22:41:34.096ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
95572025-03-25T22:41:34.096ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95582025-03-25T22:41:34.096ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95592025-03-25T22:41:34.098ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95602025-03-25T22:41:34.098ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
95612025-03-25T22:41:34.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
95622025-03-25T22:41:34.099ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 115.508986ms iteration = 79 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
95632025-03-25T22:41:34.102ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
95642025-03-25T22:41:34.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
95652025-03-25T22:41:34.102ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95662025-03-25T22:41:34.103ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
95672025-03-25T22:41:34.108ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95682025-03-25T22:41:34.108ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95692025-03-25T22:41:34.108ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
95702025-03-25T22:41:34.109ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
95712025-03-25T22:41:34.110ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
95722025-03-25T22:41:34.110ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95732025-03-25T22:41:34.112ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
95742025-03-25T22:41:34.112ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
95752025-03-25T22:41:34.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
95762025-03-25T22:41:34.113ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
95772025-03-25T22:41:34.113ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
95782025-03-25T22:41:34.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
95792025-03-25T22:41:34.114ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
95802025-03-25T22:41:34.114ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
95812025-03-25T22:41:34.115ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
95822025-03-25T22:41:34.126ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
95832025-03-25T22:41:34.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
95842025-03-25T22:41:34.131ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95852025-03-25T22:41:34.132ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
95862025-03-25T22:41:34.132ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 149.173891ms iteration = 79 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
95872025-03-25T22:41:34.133ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
95882025-03-25T22:41:34.133ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
95892025-03-25T22:41:34.133ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
95902025-03-25T22:41:34.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
95912025-03-25T22:41:34.135ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
95922025-03-25T22:41:34.135ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
95932025-03-25T22:41:34.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
95942025-03-25T22:41:34.141ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
95952025-03-25T22:41:34.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
95962025-03-25T22:41:34.147ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
95972025-03-25T22:41:34.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
95982025-03-25T22:41:34.149ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95992025-03-25T22:41:34.149ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 166.519078ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
96002025-03-25T22:41:34.154ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f7ac1017-36e7-40ec-9172-e4e1545e73c8, lookup_type: ById(f7ac1017-36e7-40ec-9172-e4e1545e73c8) } result = Ok(())
96012025-03-25T22:41:34.154ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
96022025-03-25T22:41:34.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
96032025-03-25T22:41:34.157ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 173.81208ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
96042025-03-25T22:41:34.157ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 173.502224ms iteration = 27 status = 0
96052025-03-25T22:41:34.164ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
96062025-03-25T22:41:34.164ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 180.575923ms iteration = 14 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
96072025-03-25T22:41:34.165ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96082025-03-25T22:41:34.167ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
96092025-03-25T22:41:34.167ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
96102025-03-25T22:41:34.167ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
96112025-03-25T22:41:34.172ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
96122025-03-25T22:41:34.173ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
96132025-03-25T22:41:34.173ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
96142025-03-25T22:41:34.173ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
96152025-03-25T22:41:34.173ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
96162025-03-25T22:41:34.173ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
96172025-03-25T22:41:34.173ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
96182025-03-25T22:41:34.173ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
96192025-03-25T22:41:34.173ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
96202025-03-25T22:41:34.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
96212025-03-25T22:41:34.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
96222025-03-25T22:41:34.175ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96232025-03-25T22:41:34.175ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96242025-03-25T22:41:34.177ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
96252025-03-25T22:41:34.177ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
96262025-03-25T22:41:34.177ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 193.518239ms iteration = 27 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
96272025-03-25T22:41:34.178ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
96282025-03-25T22:41:34.178ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
96292025-03-25T22:41:34.178ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
96302025-03-25T22:41:34.180ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
96312025-03-25T22:41:34.180ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
96322025-03-25T22:41:34.180ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
96332025-03-25T22:41:34.182ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
96342025-03-25T22:41:34.182ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
96352025-03-25T22:41:34.182ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 198.849947ms iteration = 14 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
96362025-03-25T22:41:34.183ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
96372025-03-25T22:41:34.183ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
96382025-03-25T22:41:34.183ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1cc4a79-f5bf-4e6f-8d56-a4c81140fff8", "content-length": "26", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} })
96392025-03-25T22:41:34.183ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
96402025-03-25T22:41:34.185ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
96412025-03-25T22:41:34.187ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
96422025-03-25T22:41:34.187ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
96432025-03-25T22:41:34.189ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 205.758752ms iteration = 27 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
96442025-03-25T22:41:34.191ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 208.072205ms iteration = 29 status = {"physical_disks_added":0}
96452025-03-25T22:41:34.194ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96462025-03-25T22:41:34.195ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
96472025-03-25T22:41:34.195ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
96482025-03-25T22:41:34.197ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96492025-03-25T22:41:34.198ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
96502025-03-25T22:41:34.198ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
96512025-03-25T22:41:34.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
96522025-03-25T22:41:34.199ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
96532025-03-25T22:41:34.199ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
96542025-03-25T22:41:34.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
96552025-03-25T22:41:34.215ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96562025-03-25T22:41:34.217ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
96572025-03-25T22:41:34.217ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
96582025-03-25T22:41:34.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
96592025-03-25T22:41:34.222ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
96602025-03-25T22:41:34.222ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
96612025-03-25T22:41:34.222ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
96622025-03-25T22:41:34.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
96632025-03-25T22:41:34.224ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96642025-03-25T22:41:34.227ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
96652025-03-25T22:41:34.227ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
96662025-03-25T22:41:34.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
96672025-03-25T22:41:34.228ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
96682025-03-25T22:41:34.228ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 0 version = 2
96692025-03-25T22:41:34.228ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
96702025-03-25T22:41:34.228ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
96712025-03-25T22:41:34.242ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
96722025-03-25T22:41:34.245ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
96732025-03-25T22:41:34.245ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 262.000969ms iteration = 27 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
96742025-03-25T22:41:34.250ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-25T22:28:31.652727Z, time_modified: 2025-03-25T22:28:31.652727Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, db_silo: Silo { identity: SiloIdentity { id: 8e66a9ad-cbc8-4793-8e94-613c5b56de7e, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-25T22:29:07.476240Z, time_modified: 2025-03-25T22:29:07.476240Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5" }] }) }
96752025-03-25T22:41:34.250ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 266.797876ms iteration = 14 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]}},"default_endpoint":{"silo_id":"8e66a9ad-cbc8-4793-8e94-613c5b56de7e","tls_certs":["41e519f5a5a2862e13a3048ffa5c65178b5d201f949832a8d8b4684248d13ae5"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
96762025-03-25T22:41:34.251ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1085 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 zone_address = fd00:1122:3344:101::2
96772025-03-25T22:41:34.251ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
96782025-03-25T22:41:34.251ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecbfccd8-e13d-4ad4-9d55-8e8530f28f96", "content-length": "26", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} })
96792025-03-25T22:41:34.251ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1088 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
96802025-03-25T22:41:34.253ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
96812025-03-25T22:41:34.259ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1085 zone_address = fd00:1122:3344:101::2
96822025-03-25T22:41:34.259ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
96832025-03-25T22:41:34.260ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "934d6612-5a06-43d8-8fae-cb539f0bdab8", "content-length": "26", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} })
96842025-03-25T22:41:34.260ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1088 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
96852025-03-25T22:41:34.261ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1122 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
96862025-03-25T22:41:34.391ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
96872025-03-25T22:41:34.391ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
96882025-03-25T22:41:34.392ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "7446588f-cb88-434c-b444-e95e401a3863", "content-length": "372", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} })
96892025-03-25T22:41:34.392ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
96902025-03-25T22:41:34.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
96912025-03-25T22:41:34.397ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96922025-03-25T22:41:34.399ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 56a83fd2-9a8c-4b6e-83cc-f31efc67765b dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
96932025-03-25T22:41:34.399ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
96942025-03-25T22:41:34.399ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-25 22:29:06.674111 UTC dns_group = internal generation = 1
96952025-03-25T22:41:34.399ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 416.013967ms iteration = 14 status = {"generation":1}
96962025-03-25T22:41:34.400ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
96972025-03-25T22:41:34.401ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "997efaf2-d482-43df-9d6d-54ec2e6b20b4", "content-length": "2", "date": "Tue, 25 Mar 2025 22:41:33 GMT"} })
96982025-03-25T22:41:34.401ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
96992025-03-25T22:41:34.401ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
97002025-03-25T22:41:34.402ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
97012025-03-25T22:41:34.403ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
97022025-03-25T22:41:34.403ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
97032025-03-25T22:41:34.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
97042025-03-25T22:41:34.426ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
97052025-03-25T22:41:34.427ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "1d80ced0-7c07-470b-b4b2-e70cae20712c", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
97062025-03-25T22:41:34.427ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
97072025-03-25T22:41:34.429ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "59e668a4-49a6-40c8-a6c5-dc8cdad3f9ba", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
97082025-03-25T22:41:34.429ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
97092025-03-25T22:41:34.432ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
97102025-03-25T22:41:34.432ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 449.216493ms iteration = 16 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
97112025-03-25T22:41:34.463ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
97122025-03-25T22:41:34.463ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
97132025-03-25T22:41:34.463ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "65785496-7120-4803-8970-e4b0da3039b2", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} })
97142025-03-25T22:41:34.463ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
97152025-03-25T22:41:34.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
97162025-03-25T22:41:34.476ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
97172025-03-25T22:41:34.477ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "bda453e2-f3d9-44b4-b22a-0e16b39582f8", "content-length": "691", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} })
97182025-03-25T22:41:34.477ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
97192025-03-25T22:41:34.477ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
97202025-03-25T22:41:34.477ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
97212025-03-25T22:41:34.479ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 965d55b6-70b6-429e-a547-a53c6610af5b dns_zone_name = oxide.test found_so_far = 1 version = 2
97222025-03-25T22:41:34.479ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
97232025-03-25T22:41:34.479ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-25 22:29:07.710706 UTC dns_group = external generation = 2
97242025-03-25T22:41:34.479ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 495.967566ms iteration = 14 status = {"generation":2}
97252025-03-25T22:41:34.479ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 496.031347ms iteration = 27 status = null
97262025-03-25T22:41:34.479ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 496.287482ms iteration = 27 status = {}
97272025-03-25T22:41:34.480ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
97282025-03-25T22:41:34.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
97292025-03-25T22:41:34.483ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
97302025-03-25T22:41:34.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
97312025-03-25T22:41:34.487ZINFOnexus (BackgroundTasks): VPC route manager sled f7ac1017-36e7-40ec-9172-e4e1545e73c8 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
97322025-03-25T22:41:34.487ZDEBGnexus (BackgroundTasks): client request SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
97332025-03-25T22:41:34.487ZDEBGnexus (BackgroundTasks): client response SledAgent = f7ac1017-36e7-40ec-9172-e4e1545e73c8 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "675879a2-e658-4d23-837a-6e6aaa06192a", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} })
97342025-03-25T22:41:34.488ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 505.158384ms iteration = 27 status = {}
97352025-03-25T22:41:34.508ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
97362025-03-25T22:41:34.509ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "6ac49f92-0056-4751-92f3-96cf009c164a", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
97372025-03-25T22:41:34.509ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
97382025-03-25T22:41:34.511ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "9b9e917d-663e-4a70-805e-e7833ecfc390", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
97392025-03-25T22:41:34.511ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
97402025-03-25T22:41:34.513ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "cf79d5bf-dda7-42d6-9ab0-1aec8ab19c4e", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
97412025-03-25T22:41:34.513ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
97422025-03-25T22:41:34.515ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
97432025-03-25T22:41:34.515ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 531.80815ms iteration = 16 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
97442025-03-25T22:41:34.518ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 535.066069ms iteration = 27 status = {}
97452025-03-25T22:41:34.577ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
97462025-03-25T22:41:34.577ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
97472025-03-25T22:41:34.579ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 13df68d7-08a4-4f56-baf6-910e4d7ba87a, rack_id: ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c) } rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
97482025-03-25T22:41:34.579ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
97492025-03-25T22:41:34.580ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
97502025-03-25T22:41:34.590ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
97512025-03-25T22:41:34.590ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cd6de27-e508-45ee-9812-41258a8f345c", "content-length": "134", "date": "Tue, 25 Mar 2025 22:41:33 GMT"} })
97522025-03-25T22:41:34.590ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1741 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
97532025-03-25T22:41:34.590ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1768 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 74d95d4e-bd7a-45f5-96cf-c73f3b7dbc1c
97542025-03-25T22:41:34.590ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
97552025-03-25T22:41:34.590ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
97562025-03-25T22:41:34.590ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
97572025-03-25T22:41:34.591ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
97582025-03-25T22:41:34.591ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
97592025-03-25T22:41:34.591ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1915 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
97602025-03-25T22:41:34.591ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
97612025-03-25T22:41:34.591ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1951 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
97622025-03-25T22:41:34.591ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9
97632025-03-25T22:41:34.591ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
97642025-03-25T22:41:34.591ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
97652025-03-25T22:41:34.593ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
97662025-03-25T22:41:34.594ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "893662fe-eb11-4d74-85f8-e5d9c60082f4", "content-length": "2", "date": "Tue, 25 Mar 2025 22:41:33 GMT"} })
97672025-03-25T22:41:34.594ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
97682025-03-25T22:41:34.594ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "da26aebd-93e9-46d9-96b1-d45d86faa37f", "content-length": "52", "date": "Tue, 25 Mar 2025 22:41:33 GMT"} })
97692025-03-25T22:41:34.594ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch_location = Switch0
97702025-03-25T22:41:34.594ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
97712025-03-25T22:41:34.701ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "5da9b4af-8100-4c04-93b2-9672f45cbec1", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} })
97722025-03-25T22:41:34.701ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 switch = Switch0
97732025-03-25T22:41:34.701ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
97742025-03-25T22:41:34.705ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = ebd9ca33-a0ad-41bc-9dc5-a9df987ca8e9 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "70f1f347-c8e3-4563-b833-7338f2446b0f", "content-length": "682", "date": "Tue, 25 Mar 2025 22:41:34 GMT"} })
97752025-03-25T22:41:34.705ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
97762025-03-25T22:41:34.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
97772025-03-25T22:41:34.710ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
97782025-03-25T22:41:34.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
97792025-03-25T22:41:34.714ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
97802025-03-25T22:41:34.714ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
97812025-03-25T22:41:34.716ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
97822025-03-25T22:41:34.717ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
97832025-03-25T22:41:34.723ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 740.185879ms iteration = 27 status = {}
97842025-03-25T22:41:36.085ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 304 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 3ac1c079-9813-478d-a53c-e3e81bc4e2da response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd
97852025-03-25T22:41:43.983ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 80 reason = Timeout
97862025-03-25T22:41:43.983ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 80 reason = Timeout
97872025-03-25T22:41:43.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
97882025-03-25T22:41:43.983ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97892025-03-25T22:41:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
97902025-03-25T22:41:43.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97912025-03-25T22:41:43.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97922025-03-25T22:41:43.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
97932025-03-25T22:41:43.988ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97942025-03-25T22:41:43.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97952025-03-25T22:41:43.994ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97962025-03-25T22:41:43.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97972025-03-25T22:41:43.996ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97982025-03-25T22:41:43.999ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b, lookup_type: ById(bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b) } result = Ok(())
97992025-03-25T22:41:43.999ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
98002025-03-25T22:41:43.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
98012025-03-25T22:41:44.009ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b original_time_created = 2025-03-25 22:28:24.513136 UTC target_id = bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b time_created = 2025-03-25 22:28:24.513136 UTC
98022025-03-25T22:41:44.009ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.477323ms iteration = 80 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"bf1240f7-9cd0-4f44-87dc-3d3cd67d5c7b","time_created":"2025-03-25T22:28:24.513136Z"}
98032025-03-25T22:41:44.014ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
98042025-03-25T22:41:44.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
98052025-03-25T22:41:44.042ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.160559ms iteration = 80 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
98062025-03-25T22:41:46.084ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 278 local_addr = [fd00:1122:3344:101::b]:40325 method = GET remote_addr = [fd00:1122:3344:101::d]:52912 req_id = 2fb809dd-62f0-4536-8fb5-0a7e10bc1cfe response_code = 200 uri = /56ee8e9a-ccae-49c2-ac33-94d5159567dd